Spark: CALL procedure for rewrite_data_files

[ajantha-bhat]

Supported the call procedure for rewrite_data_files.
Supported input arguments: table, [strategy], [sort_order], [options], [where]
Supported output arguments: rewritten_data_files_count, added_data_files_count
Added test cases to cover the scenarios.

Fixes #3355

[jackye1995]

Thanks for working on this! Almost forgot we haven't added the call procedure for this yet...

I remember Russell had laid out the arguments for the procedure in https://docs.google.com/document/d/1aXo1VzuXxSuqcTzMLSQdnivMVgtLExgDWUFMvWeXRxc, could you follow that proposal?

[ajantha-bhat]

@jackye1995 : Thanks for sharing this doc. I will look into it and implement as per that.

[ajantha-bhat]

@jackye1995 , @RussellSpitzer :
I have added some more arguments. Also updated the document.
As per me only sort related options and where arguments are pending. For where , I am not sure how to convert text to expression yet. For sort related options I am not sure whether to have as map or individual options.
If ok, I want to handle this in subsequent PR. please review the current changes.

Also I should copy the same code in spark/v3.2 folder as well ?

p.s: Also let me know if any other arguments is missed.

[RussellSpitzer]

@jackye1995 , @RussellSpitzer : I have added some more arguments. Also updated the document. As per me only sort related options and where arguments are pending. For where , I am not sure how to convert text to expression yet. For sort related options I am not sure whether to have as map or individual options. If ok, I want to handle this in subsequent PR. please review the current changes.

Also I should copy the same code in spark/v3.2 folder as well ?

p.s: Also let me know if any other arguments is missed.

For "where" can you check and see if you can produce expressions using the Spark session's parser? I think you may be able to convert to expressions using the parser and then translate them using our Spark3 utils ... I'm not sure though.

[ajantha-bhat]

@jackye1995 , @RussellSpitzer , @rdblue :
Is there any available class for converting org.apache.spark.sql.catalyst.expressions.Expression to org.apache.iceberg.expressions.Expression. I looked up based on import statement. I didn't find any.

Actually for supporting filter, I have used the spark parser to parse the string as expression. But I got spark catalyst expression. But rewrite data files action needs Iceberg expression instead of spark catalyst expression.

[rdblue]

There is a conversion from Spark to Iceberg expressions in SparkFilters.

[ajantha-bhat]

There is a conversion from Spark to Iceberg expressions in SparkFilters.

@rdblue : I think it is spark filter to Iceberg expression conversion. I needed spark expression to Iceberg expression conversion.

[ajantha-bhat]

@RussellSpitzer , @jackye1995 , @rdblue :
Please review and help merge this PR.

I will back sync to v3.0 folder once this PR is reviewed and merged. Thanks.

[ajantha-bhat]

@RussellSpitzer : So, anything else need to be done for this PR or it can be merged ?

[jackye1995]

overall looks good to me, just some nit

[ajantha-bhat]

@jackye1995 : Thanks for the review. I have pushed the nit changes also.

[RussellSpitzer]

Just had some minor comments which I think we need to wrap up, other than that I think this is looking pretty good. I know it's difficult for us to work this kind of functionality into Iceberg when Spark keeps a lot of these methods from us.

[ajantha-bhat]

@RussellSpitzer : Thanks for the review again. I have addressed the comments (also added a comment reply) and pushed the new changes.

[ajantha-bhat]

@RussellSpitzer : I have updated the test case with dataframe write instead of sql now. So, I think all comments are handled now. Thanks.

[RussellSpitzer]

Thanks so much, this is a great addition to the codebase

[ajantha-bhat]

@RussellSpitzer : Thanks for merging the PR. Appreciate your guidance and patient reply to my questions. It helped in finally avoiding the custom codes 👍🏻