System data streams are not being upgraded in the feature migration API

[masseyke]

See elastic/kibana#211614 for an example of what this looks like to the end user. We have a system data stream, .fleet-action-results that has one or more old backing indices. The user calls the feature migration API (POST /_migration/system_features), which is supposed to upgrade all system indices and data streams. That does not return any errors. Then the deprecation info API (GET /_migration/deprecations) tells them that they need to reindex the .fleet-action-results data stream (POST _data_stream/_modify). Since it is a system data stream though, the user is not allowed to reindex it. They get stuck in a position where they cannot upgrade.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[tomsonpl]

Hey team, wanted to check if there's any timeline for it already by any chance. Unfortunately the issue is stopping us from upgrading to 9.0. Thanks in advance for putting some priority on it :) cc: @dasansol92 @ferullo

[alexey-ivanov-es]

@tomsonpl I'm investigating the issue right now, but I can't provide an ETA yet. As soon as I have more information, I'll update you here

[tomsonpl]

Thanks 👍

[alexey-ivanov-es]

@tomsonpl We've identified the problem - /_migration/system_features doesn't take system data streams into account. I'm working on a fix, but it may be a big one and may take a few days to get merged