Core, Spark: Propagate orphaned delete files when rewriting data files

[nastra]

To conform to the v3 spec we're propagating orphaned DVs down to the actual APIs that rewrite/overwrite data files. That way we can ensure that there are no orphaned DVs left in delete manifests

[manuzhang]

I think we call this dangling deletes previously.

[nastra]

I previously called this rewritableDeletes but based on feedback I've renamed this to orphanedDVs()

[stevenzwu]

danglingDVs is a good idea for consistency with previous naming convention of dangling. I wasn't aware of it before.

[amogh-jahagirdar]

I'm OK either way with a slight preference for orphaned; it is inconsistent with the previous naming convention of dangling but I do think it's a much clearer name indicating the state of the DV.

[manuzhang]

Can these DVs be physically removed like orphaned files now? I think they are just removed from the current snapshot.

[stevenzwu]

LGTM.

I like Manu's naming suggestion of dangling for consistency with existing code

[amogh-jahagirdar]

Thanks @nastra on a second pass one thing I realized is we should probably double check the metadata only delete path for Spark to make sure we're cleaning up orphan DVs in that case too (where the predicates of the delete operation can completely be applied to metadata); in that case we're using the DeleteFiles API which hasn't been updated to remove the orphan DVs. In that case we'd be not cleaning up the orphaned delete files unless I'm missing something. The current changes look good in general though, just think there may be an additional case we need to address to make sure spec compliant metadata is produced.

cc @aokolnychyi @stevenzwu

[amogh-jahagirdar]

I'm OK either way with a slight preference for orphaned; it is inconsistent with the previous naming convention of dangling but I do think it's a much clearer name indicating the state of the DV.

[amogh-jahagirdar]

Nit: I know numDataFiles == numDeleteFiles in this test case but it may be clearer to just extract a separate variable for numDeleteFiles. Or just an inline comment that we expect 1 delete file per data file in this test case

[stevenzwu]

In that case too (where the predicates of the delete operation can completely be applied to metadata); in that case we're using the DeleteFiles API which hasn't been updated to remove the orphan DVs.

ok. that should be a separate PR than this.

[amogh-jahagirdar]

@stevenzwu Yeah I think I'm good with it being in a separate PR since it's a different path, but I do consider it a requirement for valid v3 metadata so it's a release blocker imo

[nastra]

Thanks @nastra on a second pass one thing I realized is we should probably double check the metadata only delete path for Spark to make sure we're cleaning up orphan DVs in that case too (where the predicates of the delete operation can completely be applied to metadata); in that case we're using the DeleteFiles API which hasn't been updated to remove the orphan DVs. In that case we'd be not cleaning up the orphaned delete files unless I'm missing something. The current changes look good in general though, just think there may be an additional case we need to address to make sure spec compliant metadata is produced.

cc @aokolnychyi @stevenzwu

Yes I'm aware of that and we won't be able to solve this with this PR. This is going to be addressed in #13222, which modifies the MergingSnapshotProducer and passes the files to be deleted to the delete manifest filter manager, which in turn will then detect orphaned DVs

[nastra]

thanks everyone for the reviews. I'll go ahead and merge this and I'll also rebase #13222 (which will address the issue when doing metadata-only deletes)