Core: Fix RESTFileScanTaskParser to handle empty delete file references list

[ajreid21]

Currently, when you deserialize the FileScanTask JSON using RESTFileScanTaskParser.fromJson, the deserializer checks whether delete_file_references node exists (but not whether it's empty), and if so, the deserializer fails at

iceberg/core/src/main/java/org/apache/iceberg/rest/RESTFileScanTaskParser.java

Line 90 in 7368e59

Collections.max(indices) < allDeleteFiles.size(),

.

TableScanResponseParser.serializeScanTasks currently always puts an empty deleted_file_references list, even if the field does not exist --

iceberg/core/src/main/java/org/apache/iceberg/rest/TableScanResponseParser.java

Line 121 in 7368e59

RESTFileScanTaskParser.toJson(fileScanTask, deleteFileReferences, spec, gen);

[nastra]

@ajreid21 the fix LGTM, can you please add a test to TestPlanTableScanResponseParser?

  @Test
  public void roundTripSerdeWithoutDeleteFiles() {
    ResidualEvaluator residualEvaluator =
        ResidualEvaluator.of(SPEC, Expressions.equal("id", 1), true);
    FileScanTask fileScanTask =
        new BaseFileScanTask(
            FILE_A,
            new DeleteFile[] {},
            SchemaParser.toJson(SCHEMA),
            PartitionSpecParser.toJson(SPEC),
            residualEvaluator);
    PlanTableScanResponse response =
        PlanTableScanResponse.builder()
            .withPlanStatus(PlanStatus.COMPLETED)
            .withFileScanTasks(List.of(fileScanTask))
            .withSpecsById(PARTITION_SPECS_BY_ID)
            .build();

    String expectedJson =
        "{\"plan-status\":\"completed\","
            + "\"file-scan-tasks\":["
            + "{\"data-file\":{\"spec-id\":0,\"content\":\"DATA\",\"file-path\":\"/path/to/data-a.parquet\","
            + "\"file-format\":\"PARQUET\",\"partition\":{\"1000\":0},"
            + "\"file-size-in-bytes\":10,\"record-count\":1,\"sort-order-id\":0},"
            + "\"residual-filter\":{\"type\":\"eq\",\"term\":\"id\",\"value\":1}}]"
            + "}";

    String json = PlanTableScanResponseParser.toJson(response);
    assertThat(json).isEqualTo(expectedJson);

    PlanTableScanResponse fromResponse =
        PlanTableScanResponseParser.fromJson(json, PARTITION_SPECS_BY_ID, false);
    PlanTableScanResponse copyResponse =
        PlanTableScanResponse.builder()
            .withPlanStatus(fromResponse.planStatus())
            .withPlanId(fromResponse.planId())
            .withPlanTasks(fromResponse.planTasks())
            .withDeleteFiles(fromResponse.deleteFiles())
            .withFileScanTasks(fromResponse.fileScanTasks())
            .withSpecsById(PARTITION_SPECS_BY_ID)
            .build();

    assertThat(PlanTableScanResponseParser.toJson(copyResponse)).isEqualTo(expectedJson);
  }

[singhpk234]

Fix, LGTM, agree with @nastra on adding the UT

[singhpk234]

The fix LGTM, i had it sitting on the part 2 pr : https://github.com/apache/iceberg/pull/13400/files#diff-584f9d53626a76efc298a57ada578de6d07c6d0b00f767f85a05e40471426374R90 since Jun 26 :)

I have one more fix which is in the part 2 pr let me get this out of this as well

[amogh-jahagirdar]

Thanks @ajreid21 @singhpk234 for this fix, agree with the fix but would be good to add the test like @nastra mentioned!

[amogh-jahagirdar]

Thanks for fixing this, we shouldn't be producing the field if there's nothing there. But we can be more accepting on the read side as done below.

[singhpk234]

LGTM too, thanks @ajreid21 !

[nastra]

thanks for the reviews @singhpk234 @amogh-jahagirdar