Flink: support split discovery throttling for streaming read #6299

stevenzwu · 2022-11-28T19:32:27Z

there are two scenarios why throttling is desirable for streaming read.

large backlog. E.g., started the streaming job from earliest snapshot
intentionally delayed consumption. E.g., if we have a stream join job reading from two Iceberg tables. First table has more real-time data and second table has typically delayed data (e.g. 24 hours). It would be great to delay the consumption for the second table by 24 hours for event time alignment. This way, we can reduce the join window in Flink state. With delayed consumption, it doesn't make sense to discovery all 24 hours of splits and hold them in enumerator memory.

stevenzwu · 2022-11-28T19:42:22Z

.../v1.16/flink/src/main/java/org/apache/iceberg/flink/source/assigner/SimpleSplitAssigner.java


  @Override
-  public GetSplitResult getNext(@Nullable String hostname) {
+  public synchronized GetSplitResult getNext(@Nullable String hostname) {


added synchronized because the new pendingSplitCount method can be called from I/O threads, while other methods are only called from the coordinator thread.

...v1.16/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/EnumerationHistory.java

…of lagging behind or intentially delayed assumption. This is to avoid eagerly discovering too many splits and tracking them in memory when the Flink job falling behind too much. It helps to keep memory footprint and enumerator checkpoint size in check.

...nk/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousIcebergEnumerator.java

...ink/src/main/java/org/apache/iceberg/flink/source/enumerator/ContinuousSplitPlannerImpl.java

bundled-guava/src/main/java/org/apache/iceberg/GuavaClasses.java

...src/test/java/org/apache/iceberg/flink/source/enumerator/TestContinuousSplitPlannerImpl.java

...rc/test/java/org/apache/iceberg/flink/source/enumerator/TestContinuousIcebergEnumerator.java

stevenzwu · 2022-12-05T21:11:41Z

Thanks @pvary and @hililiwei for the review

github-actions bot added the flink label Nov 28, 2022

stevenzwu commented Nov 28, 2022

View reviewed changes

stevenzwu requested a review from pvary November 28, 2022 20:08

stevenzwu commented Nov 28, 2022

View reviewed changes

...v1.16/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/EnumerationHistory.java Show resolved Hide resolved

stevenzwu force-pushed the throttle-split-discovery branch 2 times, most recently from b1e2beb to 8f5ba15 Compare November 29, 2022 00:32

stevenzwu force-pushed the throttle-split-discovery branch from 8f5ba15 to 6b08dbc Compare November 29, 2022 01:07

stevenzwu changed the title ~~Flink: support split discovery throttling for streaming read in case …~~ Flink: support split discovery throttling for streaming read Nov 29, 2022