Support CALL procedure for rewrite_datafiles similar to rewrite_manifests

[ajantha-bhat]

If we provide this options, It will be easy for user to handle everything from SQL itself.

https://iceberg.apache.org/spark-procedures/#:~:text=by%20this%20command-,Examples,-%C2%B6

[ajantha-bhat]

I will explore this in my free time. Please assign to me.
cc: @RussellSpitzer

[RussellSpitzer]

Great idea! I forgot we did this internally without the procedure code. So this will be a great addition!

[ajantha-bhat]

Hello @RussellSpitzer,

I have opened a draft PR for the same (#3375)
It is working with default configurations.

However I have some newbie questions as I am new to Iceberg. Please guide me.
a) Do I have to copy the same code in spark/v3.2 folder as well ? currently I have it in v3.0 folder only.
b) What is the best way to pass other arguments? I see that all the inputs has to go via TEXT form as we are using SQL for call procedure. Can I use below syntax ?

filter = Expression (Should give as SQL string and convert expression internally?)
strategy = "binpack" or "sort"
sort_order = comma separated column names [c1,c2,c3], what about asc/dsc, null_first/null_last ?

Options = comma separated "Key=value".
Example, "min-input-files=5,min-file-size-bytes=1234,max-file-size-bytes=12345,target-file-size-bytes=1024,max-file-group-size-bytes=4096,rewrite-all=true"

c) I should handle all this with testcase in current PR or can split into multiple PRs ?

[ajantha-bhat]

for b), @jackye1995 has pointed out that it is covered in the original design document (https://docs.google.com/document/d/1aXo1VzuXxSuqcTzMLSQdnivMVgtLExgDWUFMvWeXRxc).

So, I will go through it and implement accordingly.