Clean up of the `.snapshot-blob-cache*` index not working on an upgraded 8.x cluster

[pxsalehi]

Elasticsearch Version

8.x

Installed Plugins

No response

Java Version

bundled

OS Version

all

Problem Description

If a 7.x cluster has the .snapshot-blob-cache* system index (used for caching header/footer values of Lucene files when mounting searchable snapshot indices), and it is upgrade to 8.x, the clean up of the entries of this index doesn't seem to work anymore. These clean ups are done automatically periodically e.g. when the relevant searchable snapshot indices are removed (and after a configurable period). Deletion also doesn't seem to work if one would create new searchable snapshot indices (which write to this system index) and remove them. In other words, after upgrade the .snapshot-blob-cache* system index seems to only grow.

As a work-around, you can remove the .snapshot-blob-cache* system index (which requires [delete_index,manage,all] privileges). Note that this work-around essentially wipes the cache (which is what this system index is).

Steps to Reproduce

1. Start a 7.x node.
2. create an index, take a snapshot and remove the index.
3. Mount the index as a partially mounted searchable snapshot.
4. Upgrade to 8.x.
5. Remove the searchable snapshot index.
6. The .snapshot-blob-cache* index is not cleaned up after the configured retention period and periodic clean up interval.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)