[Improvement] ParallelIterable#hasNext submit reading ManifestFile task slowly with DataTableScan#planTasks

[yohengyang]

This Issuse manily to speed up reading Iterator<FileScanTask> with planTasks.

Background

We use trino to query icebergTable, but same query are sometimes very slow in scheduling stage.
Using Arthas, finally positioned to fileScanTasks.hasNext() in io.trino.plugin.iceberg.IcebergSplitSource#getNextBatch
This method traverse Iterator<FileScanTask> to get IcebergSplit .

fileScanTasks generated by DataTableScan.planTasks, when called ParallelIterable.hasNext(), it should parallel submiting Runnable task to workPool , but when called tasks.next(), it will reading manifest firstly.
When Hdfs has bad performance, this step will take too many time to stuck submitTask.
So, it's better to put reading manifest step into Runnable rather than Iterables.transform() in ManifestGroup#entries.

Through profile, we can sure that ManifestFiles.read() and ManifestReader.entries() reading avro file from the hdfs.

Reslove

I’m not familiar with iceberg. I provide a rewrite idea which may not right. Hoping everyone can help to improve it. Thanks all !
#3742

Test Performence

Query the same sql 100 times per test, the sql read 47 arvo file,
Count trino scheduling time

[liupan664021]

I met the same problem with you when using iceberg connector in presto. The ManifestFiles.read() is executed serially which can significantly slow down the file plan if manifest files are masssive. And I made similar fix as you but I haven't test the performance improvement yet. I think delaying the ManifestFiles.read() to parallel threads is more reasonable.

[yohengyang]

@liupan664021 awsome thx for suggestion~ , i make both ManifestFiles.read() and ManifestReader.entries() to parallel threads ，We have applied #3742 on our online cluster, but trino query performance did not reach the expected multiple level of improvement, I'm still doing further performance testing in trino to figure out the reason.

here's our online performence

[RussellSpitzer]

I'm confused about the task.next(), issue. From what I can see task.next should be checking the parallel iterable queue which starts by calling hasNext which repopulates the queue if it has any empty slots. It populates these slots with a runnable of the entry in the iterable.

iceberg/core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

Lines 63 to 73 in f710915

	this.tasks = Iterables.transform(iterables, iterable ->
	(Runnable) () -> {
	try (Closeable ignored = (iterable instanceof Closeable) ?
	(Closeable) iterable : () -> { }) {
	for (T item : iterable) {
	queue.add(item);
	}
	} catch (IOException e) {
	throw new RuntimeIOException(e, "Failed to close iterable");
	}
	}).iterator();

Also has anyone checked to see whether just increasing the parallelism of the work pool would work?

[yohengyang]

I'm confused about the task.next(), issue. From what I can see task.next should be checking the parallel iterable queue which starts by calling hasNext which repopulates the queue if it has any empty slots. It populates these slots with a runnable of the entry in the iterable.

iceberg/core/src/main/java/org/apache/iceberg/util/ParallelIterable.java

Lines 63 to 73 in f710915

	this.tasks = Iterables.transform(iterables, iterable ->
	(Runnable) () -> {
	try (Closeable ignored = (iterable instanceof Closeable) ?
	(Closeable) iterable : () -> { }) {
	for (T item : iterable) {
	queue.add(item);
	}
	} catch (IOException e) {
	throw new RuntimeIOException(e, "Failed to close iterable");
	}
	}).iterator();

Also has anyone checked to see whether just increasing the parallelism of the work pool would work?

@RussellSpitzer What you said is okay, There is no problem with the logic of ParallelIterable itself. The problem is that iterables is actually a TransformedIterator, which is wrapped by the ManifestGroup#entries method, and when call iterables.next, it will first call the following transform logic. The following code will go to hdfs to read the arvo file, sometimes it is very slow, thus blocking the submission

iceberg/core/src/main/java/org/apache/iceberg/ManifestGroup.java

Lines 239 to 264 in f710915

	return Iterables.transform(
	matchingManifests,
	manifest -> {
	ManifestReader<DataFile> reader = ManifestFiles.read(manifest, io, specsById)
	.filterRows(dataFilter)
	.filterPartitions(partitionFilter)
	.caseSensitive(caseSensitive)
	.select(columns);
	CloseableIterable<ManifestEntry<DataFile>> entries = reader.entries();
	if (ignoreDeleted) {
	entries = reader.liveEntries();
	}
	if (ignoreExisting) {
	entries = CloseableIterable.filter(entries,
	entry -> entry.status() != ManifestEntry.Status.EXISTING);
	}
	if (fileFilter != null && fileFilter != Expressions.alwaysTrue()) {
	entries = CloseableIterable.filter(entries,
	entry -> evaluator.eval((GenericDataFile) entry.file()));
	}
	entries = CloseableIterable.filter(entries, manifestEntryPredicate);
	return entryFn.apply(manifest, entries);

[rdblue]

It sounds like this just wants to move the transform call in ManifestGroup into the tasks that execute in parallel. That sounds reasonable to me. I'll went to take a look at this PR, but there were changes that were unrelated. Once those are removed, I'll have another look.

Thanks for working on this, @yohengyang!

[yohengyang]

#5206 makes a better implementation

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'