Core: Expire Snapshots reachability analysis #5669

amogh-jahagirdar · 2022-08-29T20:49:22Z

Currently, Snapshot expiration has a limitation which prevents file cleanup from being performed if there are multiple branches and tags. That's because for reliable file cleanup in the presence of multiple references, a reachability analysis needs to be performed to determine which files can safely be removed. The existing incremental file cleanup cannot be performed in this case.

This PR introduces a FileCleanupStrategy which is a base strategy pattern that classes override with their own file cleanup logic. There are 2 strategy implementations

1.) IncrementalFileCleanup which is used in RemoveSnapshots procedure in the case there's only 1 reference. This is simply the existing file cleanup logic.
2.) A new ReachableFileCleanup which performs a reachability analysis of reachable manifest lists, manifests, and data files given the previous and current table states.

Closes #5666.

amogh-jahagirdar · 2022-08-29T20:51:49Z

Still very incomplete, just starting a draft.

core/src/main/java/org/apache/iceberg/RemoveSnapshots.java

amogh-jahagirdar · 2022-09-04T21:53:13Z

cc: @rdblue @namrathamyske @singhpk234 @jackye1995 I moved it out of draft state, would be great to get your thoughts on this. Thanks!

core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java

core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java

core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java

amogh-jahagirdar · 2022-10-01T21:37:02Z

core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java

+    // Reads and deletes are done using Tasks.foreach(...).suppressFailureWhenFinished to complete
+    // as much of the delete work as possible and avoid orphaned data or manifest files.
+    SnapshotRef branchToCleanup = Iterables.getFirst(base.refs().values(), null);
+    if (branchToCleanup == null) {
+      return;
+    }


Porting the fix from https://github.com/apache/iceberg/pull/5666/files. This is still needed if we want to support incremental file cleanups in the case that expiration is called on a table with only non-main commits.

amogh-jahagirdar · 2022-10-01T21:39:51Z

core/src/main/java/org/apache/iceberg/FileCleanupStrategy.java

+
+  protected void deleteMetadataFiles(
+      Set<String> manifestsToDelete, Set<String> manifestListsToDelete) {
+    log.warn("Manifests to delete: {}", Joiner.on(", ").join(manifestsToDelete));


https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L538

This is still warn as it is currently, but I'm doubting if it really needs to be warn level or if we even need this logging? warn level implies that something is in a "not fatally wrong" which is not the case here. We are intentionally deleting these files after they've been correctly computed. It may mislead users. @rdblue @singhpk234 @jackye1995

rdblue · 2022-10-09T19:07:30Z

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java

+  private Set<ManifestFile> readManifests(Set<Snapshot> snapshots) {
+    Set<ManifestFile> manifestFiles = Sets.newHashSet();
+    for (Snapshot snapshot : snapshots) {
+      try (CloseableIterable<ManifestFile> manifestFilesForSnapshot = readManifestFiles(snapshot)) {


We may want to do this in parallel using Tasks. The results are copied anyway so there isn't much of a cost to it.

Oh missed this when addressing the comments, is this blocking or could I address in a follow on PR along with https://github.com/apache/iceberg/pull/5669/files#r990877250 ?

We can do it in a follow up.

In fact, we may want to do something like what you do in findFilesToDelete, where rather than reading a set of current manifest files, you just remove manifests from the candidate set. You could also use the same logic to short circuit if there are no more candidate manifests.

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java

core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java

rdblue

@amogh-jahagirdar, this looks very good. There's just one blocker here: #5669 (comment). There are some minor things as well you may want to clean up but once the blocker is in we can commit this. Thank you!

core/src/main/java/org/apache/iceberg/FileCleanupStrategy.java

zinking · 2022-10-12T06:32:15Z

core/src/main/java/org/apache/iceberg/IncrementalFileCleanup.java

+  @Override
+  @SuppressWarnings({"checkstyle:CyclomaticComplexity", "MethodLength"})
+  public void cleanFiles(TableMetadata beforeExpiration, TableMetadata afterExpiration) {
+    if (afterExpiration.refs().size() > 1) {


this method seems way too long, wasn't it better to refactor into several private methods with names explaining the intents ?

@zinking, that's a great idea, but not something we should do in this PR. This code was moved from RemoveSnapshots so we want to keep it as close to the original as possible. Would you like to follow up with a PR to simplify it and get rid of the checkstyle warning?

Yeah agreed @zinking as @rdblue mentioned, in this PR we just opted for keeping the same logic for incremental cleanup that existed in the previous RemoveSnapshots implementation. Would be great to refactor this and remove the checkstyle warning

rdblue · 2022-10-12T23:40:08Z

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java

+          .onFailure(
+              (item, exc) ->
+                  LOG.warn(
+                      "Failed to determine live files in manifest {}: this may cause orphaned data files",


This isn't true anymore. It should state that there will be a retry.

rdblue · 2022-10-12T23:44:29Z

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java

+              });
+
+    } catch (Throwable e) {
+      LOG.warn("Failed to determine the data files to be removed", e);


As a follow up, I think this message could be a little better. This makes sense if you know what the code is currently doing, but if you don't then I think it will be hard to understand that the error was in reading current manifests, rather than dropped manifests. I'd use something like "Failed to list all reachable files".

rdblue · 2022-10-12T23:45:00Z

Thanks, @amogh-jahagirdar! This looks good.

rdblue · 2022-10-12T23:45:51Z

FYI @puneetzaroo and @szehon-ho, this affects incremental table cleanup.

github-actions bot added the core label Aug 29, 2022

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch from a84f92d to 7e23bd3 Compare August 29, 2022 20:51

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 5 times, most recently from 8fcb2bc to a6807aa Compare August 30, 2022 00:11

amogh-jahagirdar commented Aug 30, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/RemoveSnapshots.java Outdated Show resolved Hide resolved

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 2 times, most recently from b9525c7 to be9232c Compare August 30, 2022 03:27

amogh-jahagirdar commented Aug 30, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/RemoveSnapshots.java Outdated Show resolved Hide resolved

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 6 times, most recently from c9e1ab8 to 302b9eb Compare September 4, 2022 05:51

amogh-jahagirdar marked this pull request as ready for review September 4, 2022 05:51

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 3 times, most recently from 324fe11 to 49021ca Compare September 4, 2022 06:04

amogh-jahagirdar commented Sep 4, 2022

View reviewed changes

core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java Show resolved Hide resolved

amogh-jahagirdar mentioned this pull request Sep 13, 2022

Performing file deletion in ExpireSnapshots with Branching and Tagging #5653

Closed

singhpk234 reviewed Sep 29, 2022

View reviewed changes

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 2 times, most recently from 95cd087 to 8698582 Compare October 1, 2022 21:32

amogh-jahagirdar commented Oct 1, 2022

View reviewed changes

rdblue reviewed Oct 9, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 9, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 9, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 9, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java Outdated Show resolved Hide resolved

rdblue reviewed Oct 9, 2022

View reviewed changes

core/src/test/java/org/apache/iceberg/TestRemoveSnapshots.java Outdated Show resolved Hide resolved

rdblue requested changes Oct 9, 2022

View reviewed changes

rdblue mentioned this pull request Oct 9, 2022

Spark: Improve performance of expire snapshot by not double-scanning retained Snapshots #3457

Merged

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 3 times, most recently from b90e8b6 to 1f73232 Compare October 10, 2022 01:53

amogh-jahagirdar commented Oct 10, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/FileCleanupStrategy.java Show resolved Hide resolved

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch from 1f73232 to df49bda Compare October 10, 2022 02:15

amogh-jahagirdar requested a review from rdblue October 10, 2022 02:23

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch 2 times, most recently from 5ba53a2 to cddc93b Compare October 11, 2022 21:33

Core: Expire Snapshots reachability analysis

b3e3a47

github-actions bot added the docs label Oct 11, 2022

amogh-jahagirdar force-pushed the expire-snapshots-reachability branch from cddc93b to b3e3a47 Compare October 11, 2022 21:34

zinking reviewed Oct 12, 2022

View reviewed changes

rdblue reviewed Oct 12, 2022

View reviewed changes

rdblue approved these changes Oct 12, 2022

View reviewed changes

rdblue merged commit cd68f9c into apache:master Oct 12, 2022

This was referenced Oct 14, 2022

Core: Parallelize the determining of reachable manifests during file cleanup #5981

Merged

Core: Assign main branch to table's current snapshot if there is no main but there is a current table state #4922

Closed

amogh-jahagirdar mentioned this pull request Oct 26, 2022

What is the expected behavior of expireOlderThan for a table with a tag that has not reached max age? #6018

Closed

zhongyujiang pushed a commit to zhongyujiang/iceberg that referenced this pull request Apr 16, 2025

[Cherry-pick] Core: Expire Snapshots reachability analysis (apache#5669)

85aecf6

Core: Expire Snapshots reachability analysis #5669

Core: Expire Snapshots reachability analysis #5669

Uh oh!

Conversation

amogh-jahagirdar commented Aug 29, 2022 • edited by rdblue Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amogh-jahagirdar commented Aug 29, 2022

Uh oh!

Uh oh!

Uh oh!

amogh-jahagirdar commented Sep 4, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amogh-jahagirdar Oct 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar Oct 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue Oct 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue commented Oct 12, 2022

Uh oh!

rdblue commented Oct 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

amogh-jahagirdar commented Aug 29, 2022 •

edited by rdblue

Loading

amogh-jahagirdar Oct 1, 2022 •

edited

Loading

amogh-jahagirdar Oct 1, 2022 •

edited

Loading

rdblue Oct 9, 2022 •

edited

Loading