[FEATURE REQUEST] The Bloom Filter for Parquet formats is necessary

[Zhangg7723]

Bloom filter in data files is useful in Iceberg reader and writer, we can also use it for delete compaction optimization, but there are no properties for the Parquet format in Iceberg, I found PRs #2582 #2642 #2643 from @jshmchenxi , these PRs are still open.
what‘s the plan about this feature? @rdblue @openinx @jshmchenxi 。

Thanks

[kbendick]

Hi @Zhangg7723! You are right that bloom filter in the data files will be useful.

It is however somewhat difficult to get right, as a lot of tuning and potentially knowledge of NDV count would need to be known ahead of time (or waste a potentially significant amount of space in the bloom filter).

I can say however, that this issue is being worked on.

@huaxingao from Apple is working on this and has reached out to the original PR author. I believe they are going to merge Apple's code with that of @jshmchenxi. The two of them would know more about it than I would, but

TLDR - This is an area of active work and not something that has been forgotten 🙂

[huaxingao]

I am collaborating with @jshmchenxi and will submit the new PR soon (hopefully today).
cc @RussellSpitzer @aokolnychyi @flyrain @szehon-ho

[kbendick]

Thank you @huaxingao! I will close this issue for now to reduce noise amongst the issues as this work is definitely on going.

Feel free to open a new issue regardings the delete compaction optimization etc @Zhangg7723, but I would suggest to wait until the new PR from Huaxing and Xi is up as anything we do will have to take that into consideration.

I know Huaxing said that hopefully the PR would be up today, but I would ask that you please give them another week or so at least.

Huaxing is really amazing at what she does (and I'm sure Xi is too), but there's always something that comes up. And open source always has extra things to consider.

But this feature is very actively being worked on and should be under review in a very reasonable time frame. This is a priority for many people. 🙂

[kbendick]

Please do reopen this issue if you think that's necessary. I'm just looking to close issues that have been addressed and I'd think any additional optimizations that need extra code should be considered after the new implementation is opened (and ideally in a new issue so as not to bug too many folks who were tagged here 😅 ).

Thank you!