Spark-3.1: support CALL procedure for rewrite_data_files #3671
Conversation
| // But these two conversions already exist and well tested. So, we are going with this approach. | ||
| SparkFilters.convert(DataSourceStrategy.translateFilter(sparkExpression, supportNestedPredicatePushdown = true).get) | ||
| } | ||
| } |
There was a problem hiding this comment.
I noticed that this Scala file was placed in the java codepath (e.g. it's in src/main/java and not src/main/scala).
Is there a reason for this?
Seems like something that might need to be updated in the original PR as well if it's not intentional.
There was a problem hiding this comment.
good catch @kbendick , I think I didn't notice it while creating a package. I will first fix in the spark-3.2 and then update these PR. Thanks
There was a problem hiding this comment.
No worries. I feel like we already have this class (though maybe I saw it when you previously merged it). I believe I saw it being used in the add_files procedure code.
If we already do, then just use that one and just remove the extra file.
Thanks for taking care of this. 🙂
There was a problem hiding this comment.
I checked add_files procedure, it doesn't involve this functionality or code. So, probably you might have seen it in my spark-3.2 merged PR.
There was a problem hiding this comment.
Also, I noticed there's a file rename here (as well as in the original PR).
I admittedly didn't review the other one and only recently became more familiar with this area of the codebase, but given the accidental Java / Scala file path thing, is this file name change intentional?
The new name of the file is: spark/v3.1/spark/src/main/java/org/apache/spark/sql/catalyst/plans.logical/SetWriteDistributionAndOrdering.scala
It's odd to me that it would be catalyst/plans.logical/SetWriteDistributionAndOrdering.scala instead of with a slash in the file path name (instead using plans.logical for just part of the path).
Maybe this is a notation I'm not familiar with? But I would definitely double check if I were you.
Thanks overall for this very welcomed additional SQL functionality 😀
@kbendick: That shows only in the PR. but when merged, github / codebase can understand that it is a sub package. |
|
CI failed because of this problem |
a7593b0 to
8868002
Compare
|
@kbendick : PR is ready. Could you please take a look at it again ? |
I have created a sub packages, instead single package. So, the path is using '/' now instead of '.' |
|
@jackye1995 : could you please add 0.13.0 milestone for this PR ? |
|
thanks for reminding, I am planning to review tomorrow morning |
|
@ajantha-bhat Are there any specific changes here from 3.2 that I should look out for? |
jackye1995
left a comment
There was a problem hiding this comment.
I did a comparison and it looks the same as 3.2, and I don't have much concern. Thanks for the backport.
|
Merged! I'll go take a look at the 3.0 backport |
This is a back sync of #3375 and its dependant #3675 to spark-3.1 folder