Update shardGenerations for all indices on snapshot finalization #128650

ywangd · 2025-05-30T05:47:52Z

If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward by updating RepositoryData and relevant in-progress snapshots so that the next finalization builds on top of this one.

Resolves: #108907

Co-authored-by: DaveCTurner [email protected]

If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward so that the next snapshot to finalize, either earlier or later in the list of entries, builts the next shardGenerations based on this one. Resolves: elastic#108907

elasticsearchmachine · 2025-05-30T05:48:18Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

elasticsearchmachine · 2025-05-30T05:48:18Z

Hi @ywangd, I've created a changelog YAML for you.

ywangd · 2025-05-30T05:53:28Z

server/src/test/java/org/elasticsearch/snapshots/SnapshotResiliencyTests.java

+            .andThen(l -> {
+                client.admin()
+                    .cluster()
+                    .prepareGetSnapshots(TEST_REQUEST_TIMEOUT, repoName)
+                    .setSnapshots(IntStream.range(0, snapshotCount).mapToObj(i -> "snapshot-" + i).toArray(String[]::new))
+                    .execute(ActionTestUtils.assertNoFailureListener(getSnapshotsResponse -> {
+                        for (final var snapshot : getSnapshotsResponse.getSnapshots()) {
+                            assertThat(snapshot.state(), is(SnapshotState.SUCCESS));
+                            final String snapshotName = snapshot.snapshot().getSnapshotId().getName();
+                            // Does not contain the deleted index in the snapshot
+                            assertThat(snapshot.indices(), contains("index-" + snapshotName.charAt(snapshotName.length() - 1)));
+                        }
+                        l.onResponse(null);
+                    }));
+            });


@DaveCTurner The test is copied from your comment. I only added this part of checking the GetSnapshots response.

I am still looking to add a randomization so that the earlier entry is a clone.

Clone is added in efdeaed

ywangd · 2025-05-30T09:55:07Z

@DaveCTurner
ConcurrentSnapshotsIT#testDeleteIndexWithOutOfOrderFinalization is failing on this PR.

The change makes a finalizing snapshot to update shard generation for earlier entry even when the index is deleted. As a result, it removes the old shard generation. But since the index is deleted, the new generation is not recorded in RepositoryData, i.e. it still refers to the old shard generation which is now deleted. This causes subsequent snapshot failure while trying to load the latest shard generation based on RepositoryData. To fix this, I tried to include the new shard generation into RepositoryData#shardGenerations. Unfortuately, this leads to more failures due to this assertion for consistent indices and shard generations in RepositoryData.

I don't think we want to remove the assertion. So the solution seems to be bringing in the concept of "shard generations for deleted indices" (e.g. something like UpdatedShardGenerations in this PR) into RepositoryData. It's doable but it is quite a lot changes including serialization. So I'd like to check with you before proceeding further. Thanks!

DaveCTurner · 2025-05-30T10:06:35Z

Ugh yeah this is tricky, but IMO it doesn't make sense to track shard generations for deleted indices in RepositoryData. If an index appears in no snapshots then it has no business being mentioned in RepositoryData. AIUI this can only be a transient situation related to snapshots happening concurrently with index deletes, so we should track it somewhere more ephemeral like SnapshotsInProgress.

ywangd · 2025-05-30T11:38:39Z

OK I think there might be a simple solution, see b478b95

The old generation got deleted not because of RepositoryData but due to SnapshotsInProgress#obsoleteGenerations. As you suggested, RepositoryData has no business with a deleted index that is not snapshotted. Since this PR updates old generation to new one for earlier entires, it must also preserve the old generation in the process since that is what recorded in RepositoryData. The new generation may end up not being used at all. That's ok, it leaves a redundant shard generation file in the repository which will be removed when snapshot deletion runs.

DaveCTurner · 2025-05-30T15:59:29Z

server/src/main/java/org/elasticsearch/repositories/FinalizeSnapshotContext.java

+     * In this case, its shard generation is tracked in {@link #deletedIndices}. Otherwise, it is tracked in
+     * {@link #liveIndices}.
+     */
+    public record UpdatedShardGenerations(ShardGenerations liveIndices, ShardGenerations deletedIndices) {


Yes I think this will work (but I will add some other comments elsewhere)

server/src/main/java/org/elasticsearch/repositories/FinalizeSnapshotContext.java

DaveCTurner · 2025-05-30T16:18:04Z

server/src/main/java/org/elasticsearch/repositories/FinalizeSnapshotContext.java

+                .entrySet()
+                .stream()
+                // We want to keep both old and new generations for deleted indices, so we filter them out here to avoid deletion.
+                // We need the old generations because they are what get recorded in the RepositoryData.


Hmm this seems suspicious. We should be updating the shard generations for existing shards in RepositoryData and discarding the previous values, even if the index isn't included in the snapshot. AIUI the tripping assertion you mentioned in an earlier comment related to generations for shards that were totally absent from RepositoryData. We should drop those shards entirely from RepositoryData, but update any ones that do exist.

Ah yes you are absolutely right. We should conditionally update the shard generations in RepositoryData for the deleted indices. The issue with my previous attempt is that the update is "unconditional" which triggered the assertion. It's now updated as suggested. Thanks a lot!

server/src/main/java/org/elasticsearch/repositories/FinalizeSnapshotContext.java

…apshotContext.java Co-authored-by: David Turner <[email protected]>

DaveCTurner

Yep I like it, just a few tiny suggestions

DaveCTurner · 2025-06-01T16:33:08Z

server/src/main/java/org/elasticsearch/repositories/blobstore/BlobStoreRepository.java

@@ -1749,7 +1749,7 @@ int sizeInBytes() {
    public void finalizeSnapshot(final FinalizeSnapshotContext finalizeSnapshotContext) {
        assert ThreadPool.assertCurrentThreadPool(ThreadPool.Names.SNAPSHOT);
        final long repositoryStateId = finalizeSnapshotContext.repositoryStateId();
-        final ShardGenerations shardGenerations = finalizeSnapshotContext.updatedShardGenerations();
+        final ShardGenerations shardGenerations = finalizeSnapshotContext.updatedShardGenerations().liveIndices();


Nit: maybe inline this, it's only used in one place, and we now need to care about which ShardGenerations we're talking about so the variable name is ambiguous.

Good point. See 1071d95

DaveCTurner · 2025-06-01T16:35:09Z

server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

                    return;
                }
                builder.put(key.index(), key.shardId(), value);
            });
        }
-        return builder.build();
+        return new UpdatedShardGenerations(builder.build(), deletedBuilder.build());


deletedBuilder.build() is mostly going to be ShardGenerations.EMPTY, can we special-case the empty collection in ShardGenerations.Builder to skip a bunch of unnecessary allocations?

Makes sense. Special cased in both Builder#build and initialization of the variable here 81f88ba

DaveCTurner · 2025-06-01T16:37:28Z

server/src/main/java/org/elasticsearch/repositories/FinalizeSnapshotContext.java

+    public record UpdatedShardGenerations(ShardGenerations liveIndices, ShardGenerations deletedIndices) {
+        public static final UpdatedShardGenerations EMPTY = new UpdatedShardGenerations(ShardGenerations.EMPTY, ShardGenerations.EMPTY);
+
+        public UpdatedShardGenerations(ShardGenerations updated) {


This constructor is only used in tests; IMO it'd be better to just be explicit and pass in ShardGenerations.EMPTY as the second parameter rather than take the risk that some future caller uses this constructor when they should be accounting for deleted indices too.

Yep see 1852486

DaveCTurner · 2025-06-01T16:42:09Z

server/src/main/java/org/elasticsearch/repositories/RepositoryData.java

@@ -424,6 +425,7 @@ public RepositoryData addSnapshot(
            // the new master, so we make the operation idempotent
            return this;
        }
+        final var shardGenerations = updatedShardGenerations.liveIndices();


Likewise this name is ambiguous; here we actually only care about the live index IDs now, maybe it'd avoid some confusion to extract that variable instead:

Suggested change

final var shardGenerations = updatedShardGenerations.liveIndices();

final var liveIndexIds = updatedShardGenerations.liveIndices().indices();

Yep fair call 6d148e5

DaveCTurner · 2025-06-01T16:43:11Z

server/src/main/java/org/elasticsearch/repositories/ShardGenerations.java

+                    }
+                }
+            });
+            return this;


IntelliJ indicates the return value is unused

Right. It is a leftover from the initial version where this method is public. It's indeed no longer needed. See a257d39

DaveCTurner · 2025-06-01T16:46:53Z

server/src/main/java/org/elasticsearch/repositories/ShardGenerations.java

@@ -244,6 +254,20 @@ public Builder put(IndexId indexId, int shardId, ShardGeneration generation) {
            return this;
        }

+        private Builder updateIfPresent(ShardGenerations shardGenerations) {
+            shardGenerations.shardGenerations.forEach((indexId, gens) -> {
+                if (generations.containsKey(indexId)) {


Could we do this with a generations.computeIfPresent() rather than a .containsKey() check followed by several .get() calls? Or even just .get() up front and then a null check?

Good point. I replaced it with a .get() and null check. I personally like it slightly better for readability than computeIfPresent. See a257d39

DaveCTurner

LGTM great stuff 🚀

ywangd · 2025-06-02T05:07:49Z

@elasticmachine update branch

elasticmachine · 2025-06-02T05:07:51Z

There are no new commits on the base branch.

elasticsearchmachine · 2025-06-02T05:10:29Z

💔 Backport failed

Status	Branch	Result
✅	8.19
❌	9.0	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 128650

ywangd · 2025-06-02T05:20:45Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Questions ?

Please refer to the Backport tool documentation

…stic#128650) If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward by updating RepositoryData and relevant in-progress snapshots so that the next finalization builds on top of this one. Co-authored-by: David Turner <[email protected]> (cherry picked from commit aa0397f) # Conflicts: # server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

…8650) (#128724) If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward by updating RepositoryData and relevant in-progress snapshots so that the next finalization builds on top of this one. Co-authored-by: David Turner <[email protected]>

…8650) (#128725) If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward by updating RepositoryData and relevant in-progress snapshots so that the next finalization builds on top of this one. Co-authored-by: David Turner <[email protected]> (cherry picked from commit aa0397f) # Conflicts: # server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java

…8650) If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward by updating RepositoryData and relevant in-progress snapshots so that the next finalization builds on top of this one. Co-authored-by: David Turner <[email protected]>

…stic#128650) If an index is deleted after a snapshot has written its shardGenerations file but before the snapshot is finalized, we exclude this index from the snapshot because its indexMetadata is no longer available. However, the shardGenerations file is still valid in that it is the latest copy with all necessary information despite it containing an extra snapshot entry. This is OK. Instead of dropping this shardGenerations file, this PR changes to carry it forward by updating RepositoryData and relevant in-progress snapshots so that the next finalization builds on top of this one. Co-authored-by: David Turner <[email protected]>

ywangd requested a review from DaveCTurner May 30, 2025 05:47

ywangd added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v9.1.0 labels May 30, 2025

elasticsearchmachine added the Team:Distributed Coordination Meta label for Distributed Coordination team label May 30, 2025

Update docs/changelog/128650.yaml

08ed60a

ywangd commented May 30, 2025

View reviewed changes

ywangd added 3 commits May 30, 2025 16:00

fix test

d0a7ab5

add clone to test

efdeaed

tweak

ce88b2c

fix test by retaining old generations

b478b95

Merge remote-tracking branch 'origin/main' into es-108907-fix

164d2f6

DaveCTurner reviewed May 30, 2025

View reviewed changes

ywangd and others added 6 commits May 31, 2025 09:03

update all the way

a75c850

tweak comment

46b8fd6

methods reorg

86fd758

Update server/src/main/java/org/elasticsearch/repositories/FinalizeSn…

8378949

…apshotContext.java Co-authored-by: David Turner <[email protected]>

spotless

9ffed9b

Merge remote-tracking branch 'origin/main' into es-108907-fix

a055214

ywangd added v8.19.0 v9.0.3 labels May 31, 2025

ywangd changed the title ~~Always update shardGenerations for previous in-progress snapshots~~ Update shardGenerations for all indices on snapshot finalization May 31, 2025

ywangd requested a review from DaveCTurner May 31, 2025 01:35

DaveCTurner reviewed Jun 1, 2025

View reviewed changes

ywangd added 6 commits June 2, 2025 09:51

inline

1071d95

optimize deletedBuilder

81f88ba

constructor always requires deleted parameter

1852486

also inline

6d148e5

more efficient update

a257d39

Merge remote-tracking branch 'origin/main' into es-108907-fix

145ff85

ywangd requested a review from DaveCTurner June 2, 2025 00:13

ywangd added the auto-backport Automatically create backport pull requests when merged label Jun 2, 2025

changelog

11b9e52

DaveCTurner approved these changes Jun 2, 2025

View reviewed changes

ywangd merged commit aa0397f into elastic:main Jun 2, 2025
18 checks passed

ywangd mentioned this pull request Jun 2, 2025

[8.19] Update shardGenerations for all indices on snapshot finalization (#128650) #128724

Merged

elasticsearchmachine added the backport pending label Jun 2, 2025

ywangd mentioned this pull request Jun 2, 2025

[9.0] Update shardGenerations for all indices on snapshot finalization (#128650) #128725

Merged

	final var shardGenerations = updatedShardGenerations.liveIndices();
	final var liveIndexIds = updatedShardGenerations.liveIndices().indices();

Update shardGenerations for all indices on snapshot finalization #128650

Update shardGenerations for all indices on snapshot finalization #128650

Uh oh!

Conversation

ywangd commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented May 30, 2025

Uh oh!

elasticsearchmachine commented May 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywangd commented May 30, 2025

Uh oh!

DaveCTurner commented May 30, 2025

Uh oh!

ywangd commented May 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

ywangd commented Jun 2, 2025

Uh oh!

elasticmachine commented Jun 2, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 2, 2025

💔 Backport failed

Uh oh!

ywangd commented Jun 2, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Uh oh!

ywangd commented May 30, 2025 •

edited

Loading