ORC: Refactor value writers #3248
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…OrcValueWriter reuses
This was referenced Oct 8, 2021
openinx
reviewed
Oct 11, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 11, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 11, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 11, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 11, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 11, 2021
openinx
reviewed
Oct 11, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 11, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcWriter.java
Outdated
Show resolved
Hide resolved
pvary
pushed a commit
to pvary/iceberg
that referenced
this pull request
Oct 11, 2021
pvary
pushed a commit
to pvary/iceberg
that referenced
this pull request
Oct 11, 2021
openinx
reviewed
Oct 12, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriters.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcWriter.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
openinx
reviewed
Oct 12, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcValueWriters.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcValueWriters.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcValueWriters.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcValueWriters.java
Outdated
Show resolved
Hide resolved
openinx
reviewed
Oct 12, 2021
spark/src/main/java/org/apache/iceberg/spark/data/SparkOrcValueWriters.java
Outdated
Show resolved
Hide resolved
Member
|
Thanks @szlta for the contribution ! The PR almost looks good to me overall, just left several comments. |
Contributor
Author
|
Thanks for the thorough review @openinx! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently there are 3 struct-like data holder types that ORC integrates with in Iceberg: Record (Iceberg), RowData (Flink) and InternalRow (Spark).
While there's an OrcRowWriter implementation for all these 3 types, only the first two relies on OrcValueWriter's, while Spark uses SparkValueWriters and that has a different API ("int ordinal" is part of nonNullWrite method). Also there's no common abstract StuctWriter class which would be similar to how Parquet/Avro writes are implemented (e.g. see https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetValueWriters.java#L539 )
Due to all that, the current ORC write code has a lot of code duplication, and fits less into Iceberg, where the write code paths for Parquet and Avro are much more in harmony.
Thus I propose to bring ORC writes up to the standard too, as we can already see some effects of this shortcoming in this PR: #2935 where a common StructWriter<?> type could help implement positional delete writer creation for ORC.