Deprecation logger rejects new entries after upgrade to 9.0 unless deprecation log data stream is rolled over

[masseyke]

Elasticsearch Version

9.0

Problem Description

In #117933, we changed the name of the dataset for the deprecation log data stream from deprecation.elasticsearch to elasticsearch.deprecation. If you had data in that data stream in 8.x or earlier, then when you upgrade to 9.0, if the deprecation logger is called the deprecations will be rejected. You will see entries like this in your (non-deprecation) logs:

Caused by: java.lang.IllegalArgumentException: [constant_keyword] field [data_stream.dataset] only accepts values that are equal to the value defined in the mappings [deprecation.elasticsearch], but got [elasticsearch.deprecation] 

[instance-0000000005] Bulk write of deprecation logs encountered some failures: [[4VK3j5UBamJ0EqvgUJmJ org.elasticsearch.index.mapper.DocumentParsingException: [1:87] failed to parse field [data_stream.dataset] of type [constant_keyword] in document with id '4VK3j5UBamJ0EqvgUJmJ'. Preview of field's value: 'elasticsearch.deprecation']]

The problem continues until you roll over the deprecation log data stream (.logs-deprecation.default). We discussed a couple of options to fix this:

1. A one-off service that gets started with a node from the deprecation xpack module. It would wait in a new feature in the cluster, and once that feature is available (so once all nodes are 9.0+), force a rollover of the .logs-deprecation.default datastream. Then the service would shut itself down.
2. Change the mapping of the dataset field to be a keyword rather than a constant_keyword, since that field is no longer a constant in this data stream (it is a constant in any backing index, but across the data stream it can now have two different values).

Steps to Reproduce

Start a cluster in 8.x. Do something that causes a deprecation message to be logged. Upgrade to 9.0. Do something that causes a deprecation message to be logged.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[dakrone]

I think the second option should be avoided, as we want to conform to ECS fields as much as possible, and constant_keyword is part of that (the newer scheme assumes a data stream only ever contains a single value for a dataset.

[masseyke]

If that's the case, we really ought to consider rolling back #117933, because that introduces a second value. If we don't roll that back, it is always possible that the data stream has two values for this field, even if we come up with a way to prevent any individual index from having no more than one value.

[dakrone]

If we roll back, we need to hold 9.0.0 for it. Otherwise if someone sets up a new 9.0.0 cluster, gets the post-#117933 value, then upgrades to 9.0.1 where it's been reverted, they'll have the same problem.