Core: Add param to limit manifest parallel reader queue size #7844

Heltman · 2023-06-15T07:10:54Z

ConeyLiu · 2023-06-16T13:35:43Z

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

 import org.apache.iceberg.relocated.com.google.common.collect.Iterables;

 public class ParallelIterable<T> extends CloseableGroup implements CloseableIterable<T> {
+  public static final int MANIFEST_READER_QUEUE_SIZE =


Why do we need this and expose as public?

ConeyLiu · 2023-06-16T13:39:38Z

core/src/main/java/org/apache/iceberg/SystemConfigs.java

+   * The size of the queue in ParallelIterable. This queue limits the memory usage of manifest
+   * reader.
+   */
+  public static final ConfigEntry<Integer> MANIFESTS_READER_QUEUE_SIZE =


The ParallelIterable is a common util, does this config is used target for manifest only?

I only found used for read manifest.

It doesn't matter that it is only used for one purpose right now. The iterable should be kept generic by passing in configuration.

ConeyLiu · 2023-06-16T13:40:40Z

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

    private final Future<?>[] taskFutures;
-    private final ConcurrentLinkedQueue<T> queue = new ConcurrentLinkedQueue<>();
+    private final LinkedBlockingQueue<T> queue =
+        new LinkedBlockingQueue<>(MANIFEST_READER_QUEUE_SIZE);


This config seems to affect all the instances.

iceberg does not seem to provide a more effective configuration management method other than environment variables. Can we turn this into a table-level configuration?

@Heltman, we typically avoid system or environment config. A more appropriate place for this is in the engine's config. Then it can pass that configuration down. For example, Flink passes threadpools into the scan API rather than using the common worker pool.

How I pass a threadpools to iceberg from trino. Trino only use plan() to get FileScanTask

Heltman · 2023-12-01T06:11:44Z

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

                                (iterable instanceof Closeable) ? (Closeable) iterable : () -> {}) {
                              for (T item : iterable) {
-                                queue.add(item);
+                                queue.put(item);


@ConeyLiu I found future.cancel can't exit this loop, because ConcurrentLinkedQueue not check InterruptedException. So we need check close in this place, to avoid memory leak.

Actural, I found trino kill query, but this loop will continue add to queue until finished.

We need use LinkedBlockingQueue or add check iterator is closed, or both them.Just like below

for (T item : iterable) { if (closed) { queue.clear() return; } queue.put(item); }

@Heltman can you please add a few more tests to TestParallelIterable that would exercise this exact condition you're describing?

Close already clears the queue and cancels tasks. I don't think that we need to modify this.

Cancels will not worker if use ConcurrentLinkedQueue. If we don't change to LinkedBlockingQueue, we need check closed every time before we add to queue.

findepi · 2023-12-15T11:29:46Z

cc @danielcweeks

danielcweeks · 2023-12-15T19:13:10Z

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

                            } catch (IOException e) {
                              throw new RuntimeIOException(e, "Failed to close iterable");
+                            } catch (InterruptedException e) {
+                              throw new RuntimeException(


Do we actually want to throw here or just log? It seems like you'll have to deal with an unnecessary/non-actionable exception when you're really just trying to cancel the iteration. Maybe just turn this into an info log message?

@danielcweeks If we had closed this iterator, we don't care of throw error or log. I don't use log only because this class don't have log, I just follow it. Maybe we can add log to this class.

danielcweeks · 2023-12-15T19:14:06Z

One minor comment but otherwise looks good to me. @nastra thoughts?

nastra

My biggest concern right now is that there aren't enough tests in TestParallelIterable that would raise the confidence in the changes being proposed

nastra · 2023-12-18T16:14:29Z

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

 import org.apache.iceberg.relocated.com.google.common.collect.Iterables;

 public class ParallelIterable<T> extends CloseableGroup implements CloseableIterable<T> {
+  public static final int ITERATOR_QUEUE_SIZE = SystemConfigs.ITERATOR_QUEUE_SIZE.value();


any particular reason to make this public? This doesn't seem to be used anywhere outside of this class. Also you probably could just use SystemConfigs.ITERATOR_QUEUE_SIZE.value() directly in L59

It's a mistake.

nastra · 2023-12-18T16:27:58Z

core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

                                (iterable instanceof Closeable) ? (Closeable) iterable : () -> {}) {
                              for (T item : iterable) {
-                                queue.add(item);
+                                queue.put(item);


@Heltman can you please add a few more tests to TestParallelIterable that would exercise this exact condition you're describing?

nastra · 2023-12-18T16:28:48Z

core/src/main/java/org/apache/iceberg/SystemConfigs.java

+   */
+  public static final ConfigEntry<Integer> ITERATOR_QUEUE_SIZE =
+      new ConfigEntry<>(
+          "iceberg.iterator.queue-size",


the config naming implies (at least to me) that this is being applied to all iterators being used in Iceberg, which isn't the case

rdblue · 2023-12-18T18:09:47Z

This class cannot use a blocking queue with the worker pool, so I'm -1 on this change.

The problem is that planning uses a shared threadpool. Using a blocking queue would cause tasks to stall, which would then tie up the threads in the shared pool and cause all planning to halt.

If you want to limit memory consumption here, then you need to do the following:

Add a new BlockingParallelIterable
Use a blocking queue in BlockingParallelIterable
Use an isolated threadpool in BlockingParallelIterable that is not shared with other planning tasks

Once that new variant is in, we can look at how to use it from the scan API. Feel free to contact me to review this, since this is a part of the code where bad changes can cause a lot of trouble!

rdblue

See my comment above.

lirui-apache · 2023-12-19T04:02:24Z

We have been using a blocking queue and shared thread pool for a while. We did hit some "dead lock" issue when running multi-stage queries with trino, because the iceberg split source generates splits synchronously. We fixed it by making it async, and so far everything works fine. I think using an isolated pool would be safer, but it may hurt performance.
Since we already have API to specify thread pool for a table scan, how about we let users deicide which pool to use? IMO whether to use blocking queue and whether to use shared pool can be configured orthogonally.

Heltman · 2023-12-19T08:56:56Z

I will add a some change for fix memory leak. And think about creating BlockingParallelIterable instead of change ParallelIterable.

findepi · 2023-12-21T20:42:27Z

The problem is that planning uses a shared threadpool. Using a blocking queue would cause tasks to stall, which would then tie up the threads in the shared pool and cause all planning to halt.

good point

3. Use an isolated threadpool in BlockingParallelIterable that is not shared with other planning tasks

from Trino perspective it would be most convenient to be able to provide an Executor.
(Trino uses shared thread pool. + a "BoundedExecutor" which limits number of threads available to a given task, while still being able to reuse threads.)

Heltman · 2024-01-03T12:17:39Z

I will add a some change for fix memory leak. And think about creating BlockingParallelIterable instead of change ParallelIterable.

I add a new pr just fix memory leak. See #9402. @rdblue

findepi · 2024-07-12T13:47:41Z

I created a PR aiming to make the queue bounded, but without requiring separate executor pool. The change is effectively transparent to class consumers. Please see #10691 and let me know what you think of that approach.

github-actions · 2024-09-03T00:13:55Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

github-actions · 2024-09-10T00:14:16Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions bot added the core label Jun 15, 2023

Heltman force-pushed the manifest-read-limit branch 2 times, most recently from 982cf30 to 45493ed Compare June 15, 2023 07:27

ConeyLiu reviewed Jun 16, 2023

View reviewed changes

Heltman force-pushed the manifest-read-limit branch from 45493ed to ac85817 Compare July 14, 2023 09:20

Heltman commented Dec 1, 2023

View reviewed changes

Heltman mentioned this pull request Dec 1, 2023

Iceberg ParallelIterable will cause coodinator a memory leak trinodb/trino#19986

Closed

Core: Add param to limit manifest parallel reader queue size

9014674

Heltman force-pushed the manifest-read-limit branch from ac85817 to 9014674 Compare December 1, 2023 06:59

danielcweeks reviewed Dec 15, 2023

View reviewed changes

nastra reviewed Dec 18, 2023

View reviewed changes

rdblue requested changes Dec 18, 2023

View reviewed changes

Heltman mentioned this pull request Jan 3, 2024

Core: Fix ParallelIterable memory leak where queue continues to be populated even after iterator close #9402

Merged

alexjo2144 mentioned this pull request Jul 10, 2024

ParallelIterable may cause memory pressure for presto coordinator #3179

Closed

findepi mentioned this pull request Jul 12, 2024

Core: Limit memory used by ParallelIterable #10691

Merged

github-actions bot added the stale label Sep 3, 2024

github-actions bot closed this Sep 10, 2024

Core: Add param to limit manifest parallel reader queue size #7844

Core: Add param to limit manifest parallel reader queue size #7844

Uh oh!

Conversation

Heltman commented Jun 15, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

findepi commented Dec 15, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielcweeks commented Dec 15, 2023

Uh oh!

nastra left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue commented Dec 18, 2023

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

lirui-apache commented Dec 19, 2023

Uh oh!

Heltman commented Dec 19, 2023

Uh oh!

findepi commented Dec 21, 2023

Uh oh!

Heltman commented Jan 3, 2024

Uh oh!

findepi commented Jul 12, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants