Spark 3.3: Add years transform function #6207
Conversation
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkYearsFunction.java
Outdated
Show resolved
Hide resolved
|
|
||
| @Test | ||
| public void testDates() { | ||
| Assert.assertEquals( |
There was a problem hiding this comment.
Just a suggestion but I prefer to also have a property style test, maybe something like
for (i in 1 to 1000) {
x = random()
assert(0, system.years(add_months(epoch, 12 * x) + system.years(add_months(epoch, -12 * x))
}
There was a problem hiding this comment.
I was inspired by our existing tests in core but let me add one more test for this too.
There was a problem hiding this comment.
After a closer look, I think current tests are easier to read as we provide the expected value explicitly.
|
|
||
| @Override | ||
| public Integer produceResult(InternalRow input) { | ||
| // return null for null input to match what Spark does in codegen |
There was a problem hiding this comment.
Makes sense, as long as this is what Spark does 👍
There was a problem hiding this comment.
Yep, that's what we decided to do in other functions such as bucket and truncate.
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestSparkYearsFunction.java
Outdated
Show resolved
Hide resolved
RussellSpitzer
left a comment
There was a problem hiding this comment.
Looks good to me, added a suggestion for some testing and there is one typo that needs to be cleaned up in a test.
nastra
left a comment
There was a problem hiding this comment.
overall LGTM, just some tiny nits/suggestions
| } | ||
|
|
||
| @Override | ||
| public Integer produceResult(InternalRow input) { |
There was a problem hiding this comment.
nit: this method appears to be the same for TimestampToYearsFunction + DateToYearsFunction, should we maybe move it to BaseToYearsFunction?
There was a problem hiding this comment.
They are slightly different. One fetches integers and the other one fetches longs from input.
There was a problem hiding this comment.
ah good call, I missed this tiny detail :)
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/functions/YearsFunction.java
Outdated
Show resolved
Hide resolved
5672568 to
fa50899
Compare
|
Thanks for reviewing, @RussellSpitzer @nastra! |
This PR adds
yearstransform function to Spark 3.3.