Fix date and timestamp transforms #1981

rdblue · 2020-12-23T21:50:30Z

This fixes incorrect values produced by date and time transforms and adds tests.

This also updates date and timestamp transform projection methods to include incorrectly transformed values.

rdblue · 2020-12-23T21:52:41Z

api/src/test/java/org/apache/iceberg/transforms/TestTimestampsProjection.java

    Expression projection = Projections.inclusive(spec).project(filter);
    UnboundPredicate<?> predicate = assertAndUnwrapUnbound(projection);

-    Assert.assertEquals(predicate.op(), expectedOp);


More tests need to be added here. I just wanted to get this up for review since I'm going to be out for a couple of days.

yyanyy

Thank you for making this change!

To make sure I understand correctly, changes in projection methods ensure that both behaviors before and after this fix will be accounted for by the projection, so that we might not need to have separate implementations for format v1 versus v2, with a slight penalty that in v2 we may scan more data than we have to?

api/src/main/java/org/apache/iceberg/transforms/Timestamps.java

api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java

api/src/test/java/org/apache/iceberg/transforms/TestDatesProjection.java

rdblue · 2020-12-24T20:56:38Z

To make sure I understand correctly, changes in projection methods ensure that both behaviors before and after this fix will be accounted for by the projection, so that we might not need to have separate implementations for format v1 versus v2, with a slight penalty that in v2 we may scan more data than we have to?

Yes, this fixes the transforms and ensures that predicate projection includes the partitions that were written with bad values. That means that we won't need different implementations for v2, but it also means that we can avoid scanning the extra partitions in the future. Because this is fixed before v2, we can ensure that all v2 tables have been fixed. So if a table is created as v2, we should be able to know that no older writers with the bug have written to the table. The only case where a v2 table would have bad metadata is when a v1 table has been converted. We should add a flag to signal that the table was converted from v1 or one that signals it was created as v2 that allows us to skip the extra partitions.

jun-he · 2021-01-06T19:59:19Z

api/src/main/java/org/apache/iceberg/transforms/Dates.java

    } else if (pred.isLiteralPredicate()) {
-      return ProjectionUtil.truncateInteger(fieldName, pred.asLiteralPredicate(), this);
+      UnboundPredicate<Integer> projected = ProjectionUtil.truncateInteger(fieldName, pred.asLiteralPredicate(), this);
+      if (this != DAY) {


Seems the same logic (if (condition) {return...} return ...) appears multiple times, can it be included inside a method, e.g. just embed it in the method defined in ProjectionUtil?

There are only 4 instance of this, and they call 2 different fix methods. I don't think it would be worth adding 2 methods just to dedup 4 lines here.

jun-he · 2021-01-06T20:04:59Z

api/src/main/java/org/apache/iceberg/transforms/Timestamps.java

+      OffsetDateTime timestamp = Instant
+          .ofEpochSecond(
+              Math.floorDiv(timestampMicros, 1_000_000),
+              Math.floorMod(timestampMicros, 1_000_000) * 1000)


nit: extract 1_000_000 and 1000 to be a constant with meaningful name or use predefined ones in java library.

RussellSpitzer · 2021-01-06T20:13:33Z

api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java

+        }
+
+        if (hasNegativeValue) {
+          return Expressions.in(projected.term(), fixedSet);


why not just always return this new expression? We build up fixedSet no matter what?

If there is no negative value, then there is no need to fixup the expression and we can return the original. That avoids some object allocation?

boroknagyz · 2021-01-08T23:58:51Z

Thanks for fixing this, I've just run into this issue with Impala. (And this fix is also a prerequisite for issue #2043)

So the plan is that 0.11+ writers will write the correct transformation values IIUC. Does it also mean that pre-0.11 readers will not be able to read the new files correctly when the data is before the epoch?

E.g. new writer writes '1969-12-31 23:05:00' with YEAR partition transform (so ts_year=-1), old reader scans for 'ts > 1969-12-31 23:00:00' (so with old transform the projected predicate is ts_year >=0), and the file will be skipped because the partition data has ts_year=-1.

I just want to clarify this, because with Impala I'm planning to write V1 files with the fixed transforms.

rdblue · 2021-01-09T00:15:55Z

Does it also mean that pre-0.11 readers will not be able to read the new files correctly when the data is before the epoch?

Yes. Older readers will still have the bug. I think we can only make a backward-compatible fix for this.

danielcweeks · 2021-01-22T00:56:50Z

So, the best I can follow, this looks good, but there's one issue I'm not clear on. If we're just expanding the projection to include possible matches (e.g. equality case), doesn't this cause a problem where the partition values are joined in (i.e. not materialized in the data like converted Hive tables)?

~~I assume we don't have a way to correct for that case since the recorded transform is incorrect and we cannot materialize the correct data to filter later?~~

Nevermind, they can't really have transforms in the partitions anyway.

danielcweeks

+1 LGTM

rdblue · 2021-01-22T01:14:27Z

Thanks for the reviews, @danielcweeks, @yyanyy, @RussellSpitzer, @jun-he, and @boroknagyz!

Because of an Iceberg bug Impala didn't push predicates to Iceberg for dates/timestamps when the predicate referred to a value before the UNIX epoch. apache/iceberg#1981 fixed the Iceberg bug, and lately Impala switched to an Iceberg version that has the fix, therefore this patch enables predicate pushdown for all timestamp/date values. The above Iceberg patch maintains backward compatibility with the old, wrong behavior. Therefore sometimes we need to read plus one Iceberg partition than necessary. Testing: * Updated current e2e tests Change-Id: Ie67f41a53f21c7bdb8449ca0d27746158be7675a Reviewed-on: http://gerrit.cloudera.org:8080/17417 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

github-actions bot added the API label Dec 23, 2020

rdblue force-pushed the add-date-and-timestamp-tests branch from d5ae0f1 to c5fe954 Compare December 23, 2020 21:51

rdblue commented Dec 23, 2020

View reviewed changes

Fix timestamp and date transforms.

a0b8816

rdblue force-pushed the add-date-and-timestamp-tests branch from c5fe954 to a0b8816 Compare December 23, 2020 21:54

yyanyy reviewed Dec 24, 2020

View reviewed changes

rdblue added 2 commits December 29, 2020 10:46

Fix checkstyle.

caad9b5

Add more tests.

b4eb6e6

rdblue requested review from aokolnychyi and danielcweeks December 29, 2020 19:47

Add docs for the projection fix methods.

146ff9e

rdblue force-pushed the add-date-and-timestamp-tests branch from a6392db to 146ff9e Compare December 29, 2020 19:51

rdblue mentioned this pull request Dec 29, 2020

Spec: document partition transforms for timestamps before 1970 #1680

Closed

rdblue added this to the Java 0.11.0 Release milestone Dec 29, 2020

rdblue added 2 commits December 29, 2020 12:12

Fix strict GT and GT_EQ projections.

8162189

Fix LT and LT_EQ strict projection.

98a9f9b

jun-he reviewed Jan 6, 2021

View reviewed changes

RussellSpitzer reviewed Jan 6, 2021

View reviewed changes

danielcweeks approved these changes Jan 22, 2021

View reviewed changes

rdblue merged commit ae96c32 into apache:master Jan 22, 2021

phd3 mentioned this pull request Mar 8, 2021

Upgrade Iceberg to 0.11.0 trinodb/trino#7049

Merged

XuQianJin-Stars pushed a commit to XuQianJin-Stars/iceberg that referenced this pull request Mar 22, 2021

API: Fix date and timestamp transforms (apache#1981)

6f1ccf4

Fix date and timestamp transforms #1981

Fix date and timestamp transforms #1981

Uh oh!

Conversation

rdblue commented Dec 23, 2020

Uh oh!

rdblue Dec 23, 2020

Choose a reason for hiding this comment

Uh oh!

yyanyy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblue commented Dec 24, 2020

Uh oh!

jun-he Jan 6, 2021

Choose a reason for hiding this comment

Uh oh!

rdblue Jan 22, 2021

Choose a reason for hiding this comment

Uh oh!

jun-he Jan 6, 2021

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer Jan 6, 2021

Choose a reason for hiding this comment

Uh oh!

rdblue Jan 6, 2021

Choose a reason for hiding this comment

Uh oh!

boroknagyz commented Jan 8, 2021

Uh oh!

rdblue commented Jan 9, 2021

Uh oh!

danielcweeks commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielcweeks left a comment

Choose a reason for hiding this comment

Uh oh!

rdblue commented Jan 22, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

danielcweeks commented Jan 22, 2021 •

edited

Loading