Flink: Support write options in FlinkSink builder #3998

hililiwei · 2022-01-28T09:45:36Z

Supports overwriting table configuration, such upsert \ overwirte\ file-format, when creating Flink sink.。

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java

stevenzwu · 2022-02-02T05:50:12Z

initially, I misread the intention was to do partition overwrite for commit

    public Builder overwrite(boolean newOverwrite) {
      this.overwrite = newOverwrite;
      return this;
    }

Can we update the PR title to sth like "Support override write configuration on top of table properties in FlinkSink builder"?

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java

rdblue

This looks close, but I think that this changes behavior when the table is set through a table loader rather than directly. This should also have tests before we can merge it. Thanks for working on this, @hililiwei!

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java

chenjunjiedada · 2022-06-22T09:36:30Z

docs/flink/flink-getting-started.md

+| write-format           | Table write.format.default | File format to use for this write operation; parquet, avro, or orc |
+| target-file-size-bytes | As per table property      | Overrides this table's write.target-file-size-bytes          |
+| upsert-enabled | Table write.upsert.enabled | Overrides this table's write.upsert.enabled |
+| overwrite | false | Overwrite the table's data |


Can we enable both upsert-enabled and overwrite? If not, should we need some doc?

nit: shall we use overwrite-enabled to keep the name consistent?

good. done. we cann't enable both upsert-enabled and overwrite

chenjunjiedada · 2022-06-22T09:49:31Z

flink/v1.15/flink/src/main/java/org/apache/iceberg/flink/FlinkWriteConf.java

+    return confParser.booleanConf()
+        .option(FlinkWriteOptions.OVERWRITE_MODE.key())
+        .flinkConfig(FlinkWriteOptions.OVERWRITE_MODE)
+        .defaultValue(false)


Should we use OVERWRITE_MODE_DEFAULT?

…erties in FlinkSink builder

hililiwei · 2022-06-25T03:20:13Z

@rdblue, Could you please take another look?

chenjunjiedada

LGTM

rdblue · 2022-06-29T14:01:46Z

docs/flink-getting-started.md

+
+| Flink option           | Default                    | Description                                                                                                |
+|------------------------| -------------------------- |------------------------------------------------------------------------------------------------------------|
+| write-file-format      | Table write.format.default | File format to use for this write operation; parquet, avro, or orc                                         |


Can you use the same options that Spark uses? That way people don't have to remember different names.

rdblue · 2022-07-03T20:09:34Z

Thanks, @hililiwei! This looks good now. I'll merge.

hililiwei · 2022-07-04T01:20:28Z

Thanks all for the reviews. I will port it to all the another versions.

optimize optimize

github-actions bot added the flink label Jan 28, 2022

kbendick reviewed Feb 1, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

stevenzwu reviewed Feb 2, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

stevenzwu reviewed Feb 3, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

hililiwei changed the title ~~Flink: support overwrite configuration in the flink sink action~~ Flink: Support override write configuration on top of table properties in FlinkSink builder Feb 7, 2022

hililiwei force-pushed the flink-sink-props branch 2 times, most recently from db3b8b0 to 36e434b Compare February 12, 2022 00:45

rdblue reviewed Feb 18, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

rdblue reviewed Feb 18, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

rdblue reviewed Feb 18, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

rdblue requested changes Feb 18, 2022

View reviewed changes

hililiwei force-pushed the flink-sink-props branch 2 times, most recently from 3f29f63 to 2ca5733 Compare February 19, 2022 08:55

hililiwei requested a review from rdblue February 21, 2022 05:58

stevenzwu reviewed Feb 23, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

stevenzwu reviewed Feb 23, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

stevenzwu reviewed Feb 23, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

hililiwei force-pushed the flink-sink-props branch 2 times, most recently from 68151c5 to 5cc1907 Compare March 1, 2022 10:52

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java Outdated Show resolved Hide resolved

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

rdblue reviewed Mar 4, 2022

View reviewed changes

flink/v1.14/flink/src/test/java/org/apache/iceberg/flink/sink/TestFlinkIcebergSink.java Outdated Show resolved Hide resolved

hililiwei force-pushed the flink-sink-props branch 2 times, most recently from d4e671b to d56f406 Compare March 8, 2022 04:35

hililiwei requested a review from rdblue March 8, 2022 06:02

chenjunjiedada reviewed Jun 22, 2022

View reviewed changes

hililiwei added 8 commits June 25, 2022 11:17

Flink 1.15: Support override write configuration on top of table prop…

27d0f9f

…erties in FlinkSink builder

in the same way as Spark.

b097079

add doc

850ceac

add doc

018e1e2

optimize

341127e

optimize

1cb12dd

optimize

24d3030

optimize

8739060

hililiwei force-pushed the flink-sink-props branch from 44f3081 to 8739060 Compare June 25, 2022 03:17

hililiwei requested a review from chenjunjiedada June 29, 2022 00:38

chenjunjiedada approved these changes Jun 29, 2022

View reviewed changes

rdblue reviewed Jun 29, 2022

View reviewed changes

rename to write-format

bfbf215

hililiwei requested a review from rdblue July 2, 2022 02:47

rdblue approved these changes Jul 3, 2022

View reviewed changes

rdblue merged commit ff6cf54 into apache:master Jul 3, 2022

hililiwei added a commit to hililiwei/iceberg that referenced this pull request Jul 4, 2022

Flink: Port apache#3998 to Flink 1.13 & 1.14

27dbc96

optimize optimize

hililiwei mentioned this pull request Jul 4, 2022

Flink: Port #3998 to Flink 1.13 & 1.14 #5194

Merged

namrathamyske pushed a commit to namrathamyske/iceberg that referenced this pull request Jul 10, 2022

Flink: Support write options in FlinkSink builder (apache#3998)

2a289b5

namrathamyske pushed a commit to namrathamyske/iceberg that referenced this pull request Jul 10, 2022

Flink: Support write options in FlinkSink builder (apache#3998)

81083a9

hililiwei added a commit to hililiwei/iceberg that referenced this pull request Jul 11, 2022

Flink: Port apache#3998 to Flink 1.13 & 1.14

f92be25

rdblue pushed a commit that referenced this pull request Jul 12, 2022

Flink: Port #3998 to Flink 1.13 & 1.14 (#5194)

eeb0af1

hililiwei added a commit to hililiwei/iceberg that referenced this pull request Oct 12, 2022

Flink: Support read options in flink source (apache#3998)

4d109e8

hililiwei deleted the flink-sink-props branch February 11, 2023 02:20

zhongyujiang added a commit to zhongyujiang/iceberg that referenced this pull request Apr 16, 2025

Flink: Port apache#3998 to 1.14.

4d55822

Flink: Support write options in FlinkSink builder #3998

Flink: Support write options in FlinkSink builder #3998

Uh oh!

Conversation

hililiwei commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stevenzwu commented Feb 2, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chenjunjiedada Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenjunjiedada Jun 22, 2022

Choose a reason for hiding this comment

Uh oh!

hililiwei Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

chenjunjiedada Jun 22, 2022

Choose a reason for hiding this comment

Uh oh!

hililiwei Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

hililiwei commented Jun 25, 2022

Uh oh!

chenjunjiedada left a comment

Choose a reason for hiding this comment

Uh oh!

rdblue Jun 29, 2022

Choose a reason for hiding this comment

Uh oh!

hililiwei Jun 30, 2022

Choose a reason for hiding this comment

Uh oh!

rdblue commented Jul 3, 2022

Uh oh!

hililiwei commented Jul 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hililiwei commented Jan 28, 2022 •

edited

Loading

chenjunjiedada Jun 22, 2022 •

edited

Loading