Flink 2.0: DynamicIcebergSink drops commits during recovery

[aiborodin]

Apache Iceberg version

1.10.0 (latest release)

Query engine

Flink

Please describe the bug 🐞

DynamicWriteResultAggregator in DynamicIcebrgSink currently produces multiple dynamic committables per table/branch/checkpoint triplet because it aggregates write results by WriteTarget, which is unique per schemaId, specId, and equality fields. It violates the idempotence contract of the DynamicCommitter, which relies on one commit request per triplet to identify and skip already committed requests during recovery.

For example, a data loss occurs in the following scenario:

1. Sink creates two commit requests with properties CheckpointID = 1, JobID = a, OperatorID = abc:

• Commit 1 (data with Schema 1)
• Commit 2 (data with Schema 2)

2. Commit requests are saved to a checkpoint
3. Committer commits Commit 1
4. Flink job restarts due to a commit failure or an autoscaling event
5. Commit requests are retrieved from a checkpoint
6. DynamicCommitter checks the Iceberg snapshots and identifies that the Committable 1 has already been committed as it matches CheckpointID = 1, JobID = a, OperatorID = abc

• DynamicCommitter skips the subsequent Committable 2, which is part of the same checkpoint

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[mxm]

Thanks for reporting this issue! I just checked the code to see how the scenario you described can occur. Let me describe what happens from the DynamicCommitter perspective in the scenario which you outlined.

Components

• DynamicWriter
• DynamicWriteResultAggregator
• DynamicCommitter

Flow

1. Checkpoint 1 gets triggered from the source. Checkpoint barrier travels through the DAG until it reaches the sink.
2. Checkpoint barrier arrives at the sink.
3. Flink's sink framework triggers DynamicWriter#prepareCommit() which sends a DynamicWriteResult downstream to DynamicWriteResultAggregator
4. Checkpoint barrier gets send downstream and arrives at DynamicWriteResultAggregator
5. DynamicWriteResultAggregator#prepareSnapshotPreBarrier(1) gets called. Manifests are written for each DynamicWriteResult and send downstream to DynamicCommitter.
6. Checkpoint barrier gets send dowstream and arrives at DynamicCommitter.
7. DynamicCommiter#commit() gets called. DynamicCommitter groups the committables by TableKey (table name, branch name) => checkpoint id => committables.
8. The maximum checkpoint id is looked up per TableKey from the flink.max-committed-checkpoint-id snapshot property. Every table/branch has its own snapshot property for the max committed checkpoint id. Committables are filtered out who are less or equal than the the max looked up checkpoint id.
9. For each TableKey (table name / branch), we commit each committable as an Iceberg table snapshot.
10. Checkpoint 1 completes.

(9) is where the flaw lies, because every committable (WriteResult) updates the snapshot properties with the the max checkpoint id. We can fix this issue by staging all WriteResults for a given TableKey and only then commit. This will also work across branches because the max checkpoint id is already maintained per branch. Even if we fail in the process, we will only attempt to commit the WriteResults for a given TableKey which haven't been committed previously.

[mxm]

(9) is where the flaw lies, because every committable (WriteResult) updates the snapshot properties with the the max checkpoint id. We can fix this issue by staging all WriteResults for a given TableKey and only then commit. This will also work across branches because the max checkpoint id is already maintained per branch. Even if we fail in the process, we will only attempt to commit the WriteResults for a given TableKey which haven't been committed previously.

There is one false assumption here. For regular append WriteResults, we can follow the above approach, but whenever delete files are present in the next WriteResult, we need to create a table snapshot for the current WriteResult, because delete files are applied before appending new data. To cover this scenario, we can add a writeResultIndex to the snapshot summary, in addition to the max chckpoint id which we already store. This allows us to skip already committed WriteResults on restore. We can optimize append-only WriteResults by putting them all into one snapshot.

[aiborodin]

but whenever delete files are present in the next WriteResult, we need to create a table snapshot for the current WriteResult, because delete files are applied before appending new data. To cover this scenario, we can add a writeResultIndex to the snapshot summary, in addition to the max chckpoint id which we already store.

@mxm let's discuss this solution in #14092.