GlueTableOperations/DynamoDbTableOperations can delete current metadata file after incorrect exception handling

[ChristinaTech]

Apache Iceberg version

1.1.0 (latest release)

Query engine

Spark

Please describe the bug 🐞

We recently encountered an issue whereby GlueTableOperations, while performing an Iceberg commit on behalf of GlueCatalog, can incorrectly interpret a successful commit as a failure, and delete the now-current table metadata file as part of cleanup. This leaves the Iceberg table inaccessible as the "current metadata pointer" now points to a deleted metadata file. We were able to correct this via an engineer manually calling Glue APIs to correct the pointer to the previous metadata file, but this represents an availability risk to our data lake service.

The reason this happens seems to be a direct result of the AWS client's default 3 attempts for a given API call, whereby Iceberg only looks at the exception thrown by the final attempt, as shown here:

org.apache.iceberg.exceptions.CommitFailedException: Cannot commit catalog_name.database_name.table_name because Glue detected concurrent update
Caused by: software.amazon.awssdk.services.glue.model.ConcurrentModificationException: Update table failed due to concurrent modifications. (Service: Glue, Status Code: 400, Request ID: <removed>)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Read timed out
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Service returned error code ServiceUnavailableException (Service: Glue, Status Code: 500, Request ID: <removed>)

We were very quickly able to determine no other writers were running on this table during the incident, which means the ConcurrentModificationException had to be from one of its own prior attempts updating the catalog despite returning a failure. If it had received the standard timeout exception, the exception logic would have correctly called checkCommitStatus and determined the commit was actually successful. However, as it only saw the ConcurrentModificationException from the final attempt, it treated the commit as failed and performed cleanup it should not have done. Notably, it would have also exhibited this incorrect behavior if the ServiceUnavailableException had been the last attempt.

As expected, Iceberg attempts to refresh its metadata and retry the commit. Unfortunately, it just deleted the object the metadata pointer directs to, resulting in:

org.apache.spark.SparkException: Writing job aborted
Caused by: org.apache.iceberg.exceptions.NotFoundException: Location does not exist: s3://fake-bucket-name/database_name.db/table_name/metadata/06814-ec5ff66c-af38-492c-ba38-55610536d9a7.metadata.json
Caused by: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: <removed>, Extended Request ID: <removed>)

While investigating, I noticed the same sequence of events would also cause DynamoDbTableOperations, which uses an AWS client configured in the same way, to take the same incorrect action, with the same outcome of the table becoming inaccessible.

Note: Removed some solution-specific information from error logs.

[ChristinaTech]

As for solutions, here are options I have considered:

Option 1: Never delete the metadata file on a failed commit

While the simplest solution, generally speaking there ARE cases where we know we can clean this up.

Option 2: Disable all retries for Glue and DynamoDB clients

This hurts reliability in normal usage to an unacceptable degree.

Option 3: Disable AWS SDK retries just for the vulnerable calls and perform retries via Iceberg Tasks class

While this mechanism can decide whether to check the actual commit status on a case-by-case basis, I have 2 concerns about this:

• It moves retry logic currently handled by the AWS SDK into Iceberg, which could cause problems for any customers providing custom AWS clients or customers not expecting the changed retry behavior.
• Every mechanism I can come up with to disable retries for specific calls is fragile and couples it to the SDK implementation and specific provides AWS Client factory.

Option 4: Detect AWS SDK retries and check commit status if they occurred

I looked into this a bit and the available ways to do this all have issues:

• Checking the thrown exception for suppressed SdkExceptions attached to it. This works but as these suppressed exceptions are not a documented part of the SDK they are vulnerable to breaking in the future.
• Attaching a metrics collector to the request specifically to grab the CoreMetrics.RETRY_COUNT value. This is publicly documented but means branching on a metric, which just feels odd. It also requires a side-channel to make this information available to the exception handler. Despite that, its probably the solution least likely to break.
• Modify the AWS SDK clients factory to provide retry count via a side-channel. This breaks if a modified AWS client factory is being provided by the end-user.

Option 5: Always check commit status if the thrown exception is an AWS SDK exception

This avoids all the retry introspection of the above approach, but means we end up spending commit time/latency checking the commit status when we might not need to. Its still preferable to risking corrupting the table state though.

Current Plans

Of these, Option 2 is easy to rule out for the reliability hit, and while Option 1 is the fastest and simplest its the most likely to leave orphan files behind. For the time being I am working on a pull request based on Option 5, as its a good starting point for further work if we go with Option 4 as well. We might decide to go with Option 3 (or an option I haven't thought of) instead, but my priority is to get a viable fix available.

[jackye1995]

cc @singhpk234 @rajarshisarkar @amogh-jahagirdar

[jackye1995]

also cc @aajisaka

[jackye1995]

Thank you for the detailed explanation and proposals.

My concern of the option is that it has performance implications, as we will take much longer to retry a commit compared to just determine based on optimistic locking failure.

I am wondering if there is a way for AWS client to stop retry if it hits service unavailable. I think we can do that through AWS client configuration, like:

GlueClient.builder()
.overrideConfiguration(ClientOverrideConfiguration.builder()
              .retryPolicy(RetryPolicy.builder()
                  .retryCondition(xxx) // some custom impl we can develop for Glue to not retry on service unavailable error or read timeout
                  .build())
              .build())

[ChristinaTech]

Yeah, having dug into a bit more and gone over #3717 since I wrote that, I'm not certain if Option 5 is the best solution anymore either as it turns a lot of cases where we really should know if we failed into UNKNOWN cases.

However, I am uncertain whether we want to tell the Glue client not to retry after a timeout or 500 error in all cases, as that could hurt the client's robustness in other scenarios. If we are gonna give it a custom retry condition anyways, it would make more sense to create an ExecutionAttribute we can attach to any individual API call we want not to retry, create a retryCondition that checks for it, and then build Option 3 using Iceberg Tasks as I described it above instead to avoid more side effects than necessary. The ExecutionAttribute and custom retryCondition can also be replaced with a second Glue Client, but that will require changes to the AWS Client factory interface, which is a public interface, so would rather avoid that.

As a secondary concern, right now, we actually treat a 500 error as a delete-the-metadata type FAILURE when the commit status after that could technically be SUCCESS or UNKNOWN. For example, its completely possible that Glue (or DynamoDB) could succeed on the backend and then error out during some post-commit step or even response marshalling.

[jackye1995]

If we are gonna give it a custom retry condition anyways, it would make more sense to create an ExecutionAttribute we can attach to any individual API call we want not to retry, create a retryCondition that checks for it

Yes this is also what I am thinking about, so if we add execution attribute only for the UpdateTable call in GlueTableOperations, we should be able to configure a retry condition for Glue client in a client factory to skip retrying when it's a 500 so it can go with tehe Iceberg Tasks lifecycle.

[ChristinaTech]

I have pulled together a rough prototype of this solution and should have a proper PR out within the week.

[ChristinaTech]

So on fleshing out to the prototype I realized it has a number of flaws:

• Ends up requiring us to recreate and control all of AWS's built in retry logic to avoid introducing any unexpected side effects.
• Effectively overrides any retry policy changes provided by a customer if they provide a custom AWS client factory.
• Requires us to merge ClientOverrideConfiguration from multiple Iceberg AWS client mutators. Also requires the same thing on the RetryPolicy object inside it.
• Doesn't end up integrating as well with the Iceberg Tasks lifecycle as originally intended.

I will put out a PR later today with an alternative solution that should have fewer potential side effects.

[ryanyuan]

hi @ChristinaTech, thanks for raising the issue and working on the PR.

We are also encountering the same issue when our KDA Flink job is trying to commit to an Iceberg table via Glue. Before your fix is in place and released, do you know if there are any workarounds or temp fixes that we could use in our code to overcome this issue?

[c0d3monk]

Hi @ChristinaTech , noticing this same issue while using apache-iceberg-0.14.0 version with Glue. Would greatly appreciate if you could suggest any workaround for that version and can a patch be expected for 0.14.x versions

[ChristinaTech]

@ryanyuan @c0d3monk So, if the issue actually occurs the way to recover the table is stated in the overview via a manual Glue UpdateTable API call to set the metadata_location property to equal the previous_metadata_location. But that's not so much as a workaround as recovery once it happens.

Considering the missing metadata failure condition is pretty easy to detect in code once it happens via catching NotFoundException, it would technically be possible to automate this fix and then retry the job, though you would have to be careful not to change anything else in the Glue Table metadata in the process to be safe.

As for a workaround that avoids ending up in this situation in the first place, some potential options besides my pending upstream fix are to:

1. Set the Catalog Option s3.delete-enabled to false so the step that actually corrupts the table becomes a no-op. If you do this though, you will Orphan any files you attempt to the system attempts to delete/expire, so make sure you don't have that option set in whatever context you use to DeleteOrphanFiles.
2. If you want to use the prior option but limit what gets Orphaned, you can potentially extend S3FileIO with a version where deleteFile no-ops for metadata files, then specify that modified version of S3FileIO for your io-impl parameter.
3. Provide a custom AWS Client factory that disables API retries for the Glue API client. The downside, as documented earlier, is:

This hurts reliability in normal usage to an unacceptable degree.

[ChristinaTech]

@c0d3monk As for whether a patch can be expected for the 0.14.x version. I can't be certain of that yet. I'd be curious to know what prevents you from updating to a more recent release.

[ryanyuan]

@ChristinaTech Thanks a lot for the suggestions.