API, Core: Add manifestLocation API to ContentFile

[amogh-jahagirdar]

This change adds a manifestPath API to ContentFile which will return the path to a manifest from which the content file resides in. If the data or delete file was not read from a manifest, the API returns null. The planned use case for this API is to enable a more optimized determination of which manifests need to be rewritten during commits.

Note: To be very clear this is not a spec change, this is an additional read API on ContentFile which surfaces the manifest path on reading of the manifest entry

Benchmark results prior to the changes:

Benchmark                                                                           Mode  Cnt   Score   Error  Units
PlanningBenchmark.distributedPlanningWithMinMaxFilter                                 ss    5   1 ± 0.160   s/op
PlanningBenchmark.localPlanningWithMinMaxFilter                                       ss    5   1 ± 0.028   s/op
PlanningBenchmark.distributedPlanningWithPartitionAndMinMaxFilter                     ss    5   0.129 ± 0.034   s/op
PlanningBenchmark.localPlanningWithPartitionAndMinMaxFilter                           ss    5   0.134 ± 0.036   s/op
PlanningBenchmark.distributedPlanningWithoutFilter                                    ss    5   2 ± 0.509   s/op
PlanningBenchmark.localPlanningWithoutFilter                                          ss    5   3 ± 0.152   s/op
PlanningBenchmark.localDataDistributedDeletesPlanningWithoutFilterWithStats           ss    5   10 ± 0.501   s/op
PlanningBenchmark.localPlanningViaDistributedScanWithoutFilterWithStats               ss    5   10 ± 0.862   s/op
PlanningBenchmark.localPlanningWithoutFilterWithStats                                 ss    5   10 ± 0.520   s/op

Benchmark results after the changes:

Benchmark                                                                           Mode  Cnt   Score   Error  Units
PlanningBenchmark.distributedPlanningWithMinMaxFilter                                 ss    5   1 ± 0.041   s/op
PlanningBenchmark.localPlanningWithMinMaxFilter                                       ss    5   1 ± 0.049   s/op
PlanningBenchmark.distributedPlanningWithPartitionAndMinMaxFilter                     ss    5   0.129 ± 0.039   s/op
PlanningBenchmark.localPlanningWithPartitionAndMinMaxFilter                           ss    5   0.126 ± 0.046   s/op
PlanningBenchmark.distributedPlanningWithoutFilter                                    ss    5   2 ± 0.654   s/op
PlanningBenchmark.localPlanningWithoutFilter                                          ss    5   3 ± 0.184   s/op
PlanningBenchmark.localDataDistributedDeletesPlanningWithoutFilterWithStats           ss    5   9 ± 0.954   s/op
PlanningBenchmark.localPlanningViaDistributedScanWithoutFilterWithStats               ss    5   10 ± 0.689   s/op
PlanningBenchmark.localPlanningWithoutFilterWithStats                                 ss    5   10 ± 0.901   s/op

[aokolnychyi]

I think going via InheritableMetadata is the easiest option.

I only have a question about CharSequence, will check the tests tomorrow.

[aokolnychyi]

Shall we put this method before pos() so both are co-located?

[aokolnychyi]

We need to discuss using CharSequence for paths in our APIs. I am not sure this approach is working out very well. I believe the idea was to expose Avro Utf8 directly. The problem is that Utf8 is not serializable. Therefore, we always convert Utf8 to String. Working with CharSequence adds a lot of complexity for no performance benefit. Puffin and a few other places use String for paths.

@rdblue, is my assumption about CharSequence correct? What are your thoughts now?

[aokolnychyi]

BTW, path() in ManifestFile returns String.

[rdblue]

I think it would be fine to use String here and always expect to convert.

[amogh-jahagirdar]

Updated to use String. I also published #11092 to deprecate the path API which returns CharSequence, in favor of a location API which returns String. This removal would happen in 2.0

[aokolnychyi]

I took another look. I am on board. My high-level comments are below.

• Use String instead of CharSequence.
• Co-locate pos() and manifestLocation() in ContentFile.
• Extend DataTableScanTestBase or whatever needed to cover distributed planning.
• Run benchmarks for distributed planning to see the impact of serializing the manifest location.

[amogh-jahagirdar]

Extend DataTableScanTestBase or whatever needed to cover distributed planning.
Run benchmarks for distributed planning to see the impact of serializing the manifest location.

Great points on these, will run distributed planning benchmarks to make sure there's not a regression from adding this.

[amogh-jahagirdar]

@aokolnychyi I ran the plan benchmarks, and added to the PR description. there's no significant difference in planning before and after.

Extend DataTableScanTestBase or whatever needed to cover distributed planning.

For tests, we actually already have SparkDistributedDataScanTestBase which tests different combinations of local/distributed for data file and delete file planning. We have TestSparkDistributedDataScanJavaSerialization and TestSparkDistributedDataScanKryoSerialization for testing java/kryo serialization respectively. So I think we're effectively getting coverage in different planning + serialization combinations for the new code path. Please let me know if you think there's a coverage gap though!

[aokolnychyi]

LGTM, one nit in the tests. Thanks, @amogh-jahagirdar!

[amogh-jahagirdar]

Verified TestSparkReaderDeletes#testMultiplePosDeleteFiles() failure is unrelated (ran locally a few times with my changes and it passed). The test may be flaky. I'll push another change to address the nit and we can verify if it's flaky.

TestSparkReaderDeletes > testMultiplePosDeleteFiles() > format = orc, vectorized = false, planningMode = DISTRIBUTED FAILED
    java.lang.AssertionError: [Metric values were not finalized] 
    Expecting actual not to be null
        at org.apache.iceberg.spark.source.SparkSQLExecutionHelper.lastExecutedMetricValue(SparkSQLExecutionHelper.java:63)
        at org.apache.iceberg.spark.source.TestSparkReaderDeletes.deleteCount(TestSparkReaderDeletes.java:205)

[amogh-jahagirdar]

This time TestDataFrameWrites > testFaultToleranceOnWrite() failed which is known to be flaky: #10811

[amogh-jahagirdar]

I'm going to go ahead and merge. Thanks for reviewing @aokolnychyi @rdblue!