Spark: Detect dangling DVs properly

[nastra]

No description provided.

[singhpk234]

file-paths with s3://bucket/.parquet and s3a://bucket/.parquet would be still the same file, it would nice if we can handle these scenario as well like we do in orphan file clean up
#4346 (comment)

[nastra]

sorry I'm not fully following your comment. Can you please elaborate?
We mainly care about DVs here (that are puffin files) which point to non-existing data files

[singhpk234]

Apologies for the confusion, this comment was meant to be in the line below, essentially where we matching the data file path with the file path puffin is pointing to.

can having an exact equality check lead to miss ? for ex consider in the table if file_path 's3://<tbl_location>/filea.parquet' exists but Puffin files point to 's3a://<tbl_location>/filea.parquet' since we do exact not eq check this case can be missed as only diff is S3 and S3a but the file is there ?

Hence was recommending the above

[danielcweeks]

I think @singhpk234 is pointing out the potential for scheme mismatch. The handling done in orphan files is here

We might be able to normalize the schemes as part of the projection prior to join in order to avoid these issues.

[nastra]

I'm not sure this is actually an issue. Throughout the codebase the DeleteFile#referencedDataFile() is always referring to a DataFile's location, such as here:

iceberg/core/src/main/java/org/apache/iceberg/DeleteFileIndex.java

Line 498 in 5bd314b

String path = dv.referencedDataFile();

Also when the DeleteFile is created, it always references the location of the DataFile:

iceberg/core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java

Line 130 in f186be7

.withReferencedDataFile(referencedDataFile)

[singhpk234]

I agree, and this is not a blocker, just brought up to see if we can handle this. While this may not be an issue for java impl for iceberg DV but can this happen or be potentially missed in other language implementation. If we put this in a diff way like this, does iceberg consider a file with 's3://<tbl_location>/data-filea.parquet' and 's3a://<tbl_location>/data-filea.parquet' same, should we consider they are handled same way through out ?

[danielcweeks]

+1 @nastra my only concern is to double check that we don't physically delete the file after removing it. That would be safe for PosDel files but not DVs. I couldn't find a place where that was happening, but we should make sure that there's nothing in the snapshot expiration or other paths that physically remove the files after we mark the entries deleted.

[nastra]

+1 @nastra my only concern is to double check that we don't physically delete the file after removing it. That would be safe for PosDel files but not DVs. I couldn't find a place where that was happening, but we should make sure that there's nothing in the snapshot expiration or other paths that physically remove the files after we mark the entries deleted.

thanks for the review @danielcweeks. I'll double check this and will follow up in a separate PR on this