Use particular worker pool for flink jobs #4177

yittg · 2022-02-21T06:49:48Z

yittg · 2022-02-22T05:16:25Z

@rdblue @openinx Please help review this change.

core/src/main/java/org/apache/iceberg/util/ThreadPools.java

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java

rdblue · 2022-02-22T17:38:37Z

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/source/StreamingMonitorFunction.java

  }

+  @Override
+  public void open(Configuration parameters) throws Exception {


Should the pool size be configured by parameters?

Also, is there a way to share pools if there are multiple Iceberg operators in the same Flink job?

Should the pool size be configured by parameters?

Configured from scan context;

Also, is there a way to share pools if there are multiple Iceberg operators in the same Flink job?

I think it's hard to share, and it will easily get meaningless across distributed nodes.
What do you think, @rdblue?

@rdblue do you think that sharing a pool in one job is a blocking issue? If that, we can provide a pool keyed by job. it's somehow reasonable to replace the original pool equivalently :)

Hey, sorry about this. I think my comment here is probably what caused the confusion about sharing pools by job ID. I think there are use cases around this (Steven has one at least) but let's focus on fixing the problem here and sharing resources later.

Thanks for your patience, @yittg!

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java

stevenzwu · 2022-03-10T20:56:40Z

core/src/main/java/org/apache/iceberg/util/ThreadPools.java

    return WORKER_POOL;
  }

+  public static ExecutorService newWorkerPool(String namePrefix, Integer parallelism) {


nit: is poolSize more intuitive than parallelism?

stevenzwu · 2022-03-10T20:57:17Z

core/src/main/java/org/apache/iceberg/util/ThreadPools.java

+  public static ExecutorService newWorkerPool(String namePrefix, Integer parallelism) {
+    return MoreExecutors.getExitingExecutorService(
+        (ThreadPoolExecutor) Executors.newFixedThreadPool(
+            Optional.ofNullable(parallelism).orElse(WORKER_THREAD_POOL_SIZE),


should we make the param a primitive type and provide an overload method without the poolSize param?

I'm okay either way.

stevenzwu · 2022-03-10T21:04:03Z

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/source/StreamingMonitorFunction.java

+    super.open(parameters);
+
+    final String jobId = getRuntimeContext().getJobId().toString();
+    this.workerPool = ThreadPools.newKeyedWorkerPool(jobId, "flink-worker-pool", scanContext.planParallelism());


I saw this shares the same key as IcebergFilesCommitter, but not FlinkInputFormat. Trying to understand the reasons.

I agree here. Since this is creating a different thread pool per job ID, the thread name prefix should also include the job ID to get unique names.

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/source/SplitHelpers.java

rdblue · 2022-03-10T21:28:23Z

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/source/StreamingMonitorFunction.java

+    final String jobId = getRuntimeContext().getJobId().toString();
+    this.workerPool = ThreadPools.newKeyedWorkerPool(jobId, "flink-worker-pool", scanContext.planParallelism());
+    getRuntimeContext().registerUserCodeClassLoaderReleaseHookIfAbsent(
+        "release-flink-worker-pool", () -> ThreadPools.shutdownKeyedWorkerPool(jobId));


Is the key here also going to be a problem? Or is this a description?

rdblue · 2022-03-10T21:30:10Z

core/src/main/java/org/apache/iceberg/util/ThreadPools.java

+                .build()));
+  }
+
+  public static ExecutorService newKeyedWorkerPool(String key, String namePrefix, Integer parallelism) {


I don't think that we need to keep worker pools here in a static map.

The two places where this is called immediately set up a callback that calls shutdown, but could easily keep a reference to the worker pool locally instead of storing it here by name.

I think it would be better to avoid keeping track of pools here.

if the intention is to reuse the job specific thread pool in Flink, then we do need the static cache as the same keyed pool may be requested from multiple code path.

Is this a Flink only problem regarding classloader issue on thread pool? if so, maybe we can move the keyed cache into Flink module.

Oh, so the job can share between the monitor and the sink? I don't really mind having two pools for that.

@rdblue ,sorry, i don't get your point exactly. let me guess, what you really mean is sharing pools for all sources or all sinks, not for all sources and sinks?
To be clear, for example, if a job consists of:
Source: Iceberg A(parallelism: 3), Source:Iceberg B, Sink:Iceberg C, Sink: Iceberg D.
What's your favor?

share btw all parallelism of one operator, like 3 subtask for Iceberg A (it can be run in different slots in one TaskManager or different TaskManagers) ;

share btw all sources or all sinks, like sharing one for btw A and B, and another one for C and D;

share btw all operators, like sharing btw A, B, C, and D; all subtasks in same TaskManager can share.

What I was thinking was a pool per operator in a job, rather than a pool per job. That avoids the need to track thread pools by some key in static state. I think it is probably fine to have more pools since these are primarily for IO. Does that sound reasonable?

after reviewing the usage of the thread pools, I am also in favor of no sharing of thread pools so that we can avoid the static cache. None of the usage is on parallel tasks.

source: split planning (running on jobmanager or the single-parallelism StreamingMonitorFunction)

sink: single-parallelism committer

But we do need to add some user doc to clarify the behavior change regarding I/O thread pool. Previously, there is a global shared thread pool per JVM. Now it is per source/sink. E.g., Internally we had a rather unique setup where a single Flink job (running on many taskmanagers) can ingest data to dozens or hundreds of Iceberg tables. For those setups, users would need to tune down the pool size to probably 1 to avoid excessive number of threads created in JVM.

@stevenzwu, for that use case, maybe we should follow up to this PR with one that allows you to configure a named threadpool? I think that's probably the use case that @yittg had in mind when he set up sharing.

In addition to documentation change, we should also make sure this behavior change is captured in the release note of the next minor version release of 0.14.0. @rdblue where do we track future release note?

@stevenzwu, I added the "release notes" tag to this PR and added it to the 0.14.0 release milestone so we add this to release notes. If you want, you can add a comment with the suggested release notes at the end.

Yeah, although I didn't think about sharing at the beginning. but some kind of sharing or global limit sounds good to me after some consideration. We can provide a reasonable solution next I think.
Thanks, @rdblue and @stevenzwu .

rdblue

Thanks, @yittg! I really appreciate how patient you've been with me getting back to this review.

There are two main things to fix now. First, I don't think we need to keep track of open pools in ThreadPools. Second, I agree with @stevenzwu's comment about passing the same name prefix for all of the pools created by the monitor and the sink. We should make sure the prefix is also unique by job ID.

Thanks!

yittg · 2022-03-11T08:38:38Z

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/source/FlinkInputFormat.java

+    final ExecutorService workerPool = ThreadPools.newWorkerPool("iceberg-plan-worker-pool", context.planParallelism());
    try (TableLoader loader = tableLoader) {
      Table table = loader.loadTable();
-      return FlinkSplitPlanner.planInputSplits(table, context);
+      return FlinkSplitPlanner.planInputSplits(table, context, workerPool);
+    } finally {
+      workerPool.shutdown();


This function is called in client and job manager. So there is no context here. Given it's a adhoc pool and will be shut down after planning, i think it's ok to name it in this way.

yittg

Given the StreamingMonitorFunction and IcebergFilesCommitter are both 1-parallelism. We can new worker pool in subtask open along with guaranteeing one worker pool per operator.

github-actions bot added API core flink labels Feb 21, 2022

yittg changed the title ~~Use particuar worker pool for flink jobs~~ Use particular worker pool for flink jobs Feb 21, 2022

yittg mentioned this pull request Feb 21, 2022

Fixed context classloader of threads in ThreadPools #3797

Closed

yittg force-pushed the flink-class-loader-closed branch from 93762c7 to 82ea9c1 Compare February 22, 2022 01:53

yittg marked this pull request as ready for review February 22, 2022 03:43

rdblue reviewed Feb 22, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/util/ThreadPools.java Outdated Show resolved Hide resolved

rdblue reviewed Feb 22, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java Show resolved Hide resolved

rdblue reviewed Feb 22, 2022

View reviewed changes

yittg force-pushed the flink-class-loader-closed branch from 8279039 to 6b7666a Compare February 23, 2022 05:44

yittg commented Feb 23, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergFilesCommitter.java Outdated Show resolved Hide resolved

yittg force-pushed the flink-class-loader-closed branch 2 times, most recently from 1289e7c to 6af99ed Compare February 23, 2022 07:39

yittg requested a review from rdblue February 24, 2022 01:57

yittg force-pushed the flink-class-loader-closed branch 2 times, most recently from 45f3de2 to 8f3733a Compare March 1, 2022 04:17

stevenzwu reviewed Mar 10, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/source/SplitHelpers.java Show resolved Hide resolved

rdblue reviewed Mar 10, 2022

View reviewed changes

rdblue requested changes Mar 10, 2022

View reviewed changes

Use particuar worker pool for flink jobs

250d8e4

yittg force-pushed the flink-class-loader-closed branch from 8f3733a to 62b04d9 Compare March 11, 2022 08:33

yittg commented Mar 11, 2022

View reviewed changes

yittg requested a review from rdblue March 11, 2022 08:39

yittg commented Mar 11, 2022

View reviewed changes

Support configure parallelism

efcd332

yittg force-pushed the flink-class-loader-closed branch from 62b04d9 to efcd332 Compare March 11, 2022 09:18

rdblue approved these changes Mar 11, 2022

View reviewed changes

rdblue merged commit 1b774c2 into apache:master Mar 11, 2022

rdblue added this to the Iceberg 0.14.0 Release milestone Mar 11, 2022

rdblue added the release notes PR should be included in the release notes label Mar 11, 2022

yittg deleted the flink-class-loader-closed branch March 12, 2022 03:11

Use particular worker pool for flink jobs #4177

Use particular worker pool for flink jobs #4177

Uh oh!

Conversation

yittg commented Feb 21, 2022

Uh oh!

yittg commented Feb 22, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevenzwu Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

yittg Mar 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yittg left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stevenzwu Mar 11, 2022 •

edited

Loading

yittg Mar 11, 2022 •

edited

Loading