Core: Use V2 format in new tables by default #8381

aokolnychyi · 2023-08-23T18:49:31Z

This PR switches new tables to use the V2 format by default, as discussed on the dev list.

The purpose of this change is to avoid some workarounds in the V1 format that may impact the user experience for anyone starting with Iceberg. In particular, v1 tables use always null transforms during spec evolution (confusing) and require a flag to enable snapshot ID inheritance (performance).

Note that V2 tables without delete files (disabled by default) should be no different than V1 tables. If there are engines that don't support V2 tables, they should modify their logic to check for presence of delete files instead.

rdblue · 2023-08-23T19:03:44Z

core/src/main/java/org/apache/iceberg/TableMetadata.java


    return new Builder()
-        .upgradeFormatVersion(formatVersion)
+        .setFormatVersion(formatVersion)


I think it's a bit dangerous to expose the ability to set rather than upgrade the format version. I think people are going to call it when they shouldn't. Is it actually needed here?

The builder inits the format version to the default format version (2 now) and if users explicitly pass 1 as the desired format version, create table statements start to fail as it is not allowed to downgrade.

We can make this package-private given that it is only used during table creation. Or we could lazily init the default format version. Either way would work for me.

I made it private, which is still accessible in TableMetadata. It is kind of weird to call upgrade when creating a fresh metadata file.

I feel like upgradeFormatVersion (despite the name) might be more consistent the other setters, which perform checks and add the MetadataUpdate as needed.

I don't think the validation in upgradeFormatVersion is applicable when creating new tables. I also don't think we are supposed to generate an update cause we are not updating the format version, the table does not exist.

aokolnychyi · 2023-08-23T21:39:13Z

@rdblue @nastra @Fokko @amogh-jahagirdar, could you help debug fails in TestRESTCatalog? It is the only catalog test that fails in core. It seems the data sequence number for snapshot is not regenerated with concurrent operations.

bryanck · 2023-08-23T21:55:55Z

When testing w/ V2 as the default, I encountered a REST catalog test failure with the concurrent replace transaction tests. in particular this check was failing. Say transaction 1 starts, then transaction 2 starts and commits, then transaction 1 commits, the sequence number for the second commit will be the same as the first. One such test is this one. It seems like concurrent transactions will always fail with V2 because of this. (Just saw your post now after I posted this @aokolnychyi )

@nastra I think you found the issue to be with CatalogHandlers rather than RESTTableOperations so more of a catalog implementation issue?

aokolnychyi · 2023-08-23T22:39:29Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java

    table.refresh();
    PartitionSpec expected =
-        PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts").build();
+        PartitionSpecParser.fromJson(


I had to switch to JSON so that I can correctly validate field IDs used in the spec, which is not accessible in a programatic way in this module.

bryanck · 2023-08-23T23:21:35Z

@nastra @amogh-jahagirdar We may need to update the REST catalog commit protocol for this. When we pass the UpdateTableRequest, a catalog service implementation will base the update on the current metadata. There isn't enough information in the request to use a different (earlier) base I don't think.

aokolnychyi · 2023-08-24T19:37:47Z

@bryanck @nastra @amogh-jahagirdar, I am wrapping up required test changes. Do you think it would be possible to fix the REST catalog tests/implementation quickly or shall I keep using V1 tables there for now?

bryanck · 2023-08-24T19:42:13Z

@aokolnychyi I feel like the REST catalog change will need to happen in a follow-up, so I'd say keep with V1 for now there if you want to get this in.

aokolnychyi · 2023-08-24T21:16:18Z

@bryanck, sounds good. I created #8390 to track the fix. Meanwhile, I ignored failing tests for REST catalogs and referenced the issue. I couldn't use V1 tables there cause our test REST implementation complained about format overrides (seems like another issue to look into).

nastra · 2023-08-25T16:35:28Z

spark/v3.4/spark-runtime/src/integration/java/org/apache/iceberg/spark/SmokeTest.java

-    Assert.assertTrue(
-        "All transforms should be void",
-        table.spec().fields().stream().allMatch(pf -> pf.transform().toString().equals("void")));
+    Assert.assertTrue("Table should be unpartitioned", table.spec().isUnpartitioned());


minor: I think it would be good to switch assertions to AssertJ in such a case - the project is trying to move away from Junit4-style assertions, and it would be great to have new assertions written using AssertJ to make the migration process easier

Yeah, I've used the new assertion style whenever adding new checks or when modifying classes that migrated. Here, I modify the existing check so I followed the rest of the class. I was not sure it is worth to just switch a single check.

aokolnychyi · 2023-08-25T17:32:46Z

@bryanck @szehon-ho @rdblue @nastra @RussellSpitzer, this one is ready for a detailed review round. Could you take a look? This change covers many modules and rebasing it won't be fun.

RussellSpitzer · 2023-08-25T19:58:13Z

core/src/main/java/org/apache/iceberg/TableMetadata.java


+    // it is only safe to set the format version directly while creating tables
+    // in all other cases, use the method below to safely upgrade the format version
+    private Builder setFormatVersion(int newFormatVersion) {


The doc comment here is confusing since the method is write below here, I think you should say "in all other cases, use upgradeFormatVersion.

We also could try to encapsulate some of this doc into the method name? newTableFormatVersion?

I changed the method name and the doc.

RussellSpitzer · 2023-08-25T20:06:25Z

core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java

+            .withProperty("key1", "value1")
+            .createOrReplaceTransaction();
+
+    createTxn.newAppend().appendFile(FILE_A).commit();


nit: but we could refactor all this out since it's all identical except for the version. Not a big deal now, but maybe makes sense since we'll have a "V3" soon enough

I refactored all applicable tests to run against both versions.

RussellSpitzer · 2023-08-25T20:07:12Z

core/src/test/java/org/apache/iceberg/jdbc/TestJdbcCatalog.java

  }

+  @Test
+  public void testReplaceTxnBuilderDefaultFormatVersion() {


Same nit about extracting out common code since we will probably have 3 of these tests in the near future.

RussellSpitzer · 2023-08-25T20:08:35Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java

-            .build();
-
-    Assert.assertEquals("Should have new spec field", expected, table.spec());
+    Assert.assertTrue("New spec must be unpartitioned", table.spec().isUnpartitioned());


Do we want to keep a test for the V1 Behavior? We haven't actually deprecated it.

These are being tested against both now.

RussellSpitzer · 2023-08-25T20:12:51Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java

    sql("ALTER TABLE %s ADD PARTITION FIELD days(ts)", tableName);
    table.refresh();
    PartitionSpec expected =
-        PartitionSpec.builderFor(table.schema()).withSpecId(1).day("ts").build();


We do we need to do the change here? I'm not a big fan of have String constants for expected values, even if we are parsing them into a friendly object for comparison.

I mentioned the reason why some tests refer to String constants here.

RussellSpitzer · 2023-08-25T20:13:58Z

spark/v3.1/spark/src/test/java/org/apache/iceberg/spark/actions/TestCreateActions.java

    createSourceTable(CREATE_PARQUET, source);
-    assertSnapshotFileCount(SparkActions.get().snapshotTable(source).as(dest), source, dest);
+    SnapshotTable action = SparkActions.get().snapshotTable(source).as(dest);
+    action.tableProperty(TableProperties.FORMAT_VERSION, "1");


Does this test need to be with Version1, or is this just to minimize changes

This actually has to be v1 as the test verifies it is v1 and then does an update to v2.

Makes sense, didn't get that from the test name

RussellSpitzer

Left a few comments. I think this is a good thing to do but it seems like we have taken the approach of fixing each test suite in the way that is simplest. Those which are more complicated are kept on V1 and others are moved to V2. I think it's fine if we want to follow up with this later but we probably need to decide if we want to:

Do all the tests on the default spec version
Do all tests with all available spec versions

This is a bit of a mix at the moment. I do think this would be a good task though for a new contributor.

aokolnychyi · 2023-08-28T23:57:58Z

core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java


-  @Test
-  public void testCreateTableBuilder() throws Exception {
+  @ParameterizedTest


It did not make sense to parameterize the entire suite, so I only changed applicable tests.

RussellSpitzer · 2023-08-30T17:14:43Z

core/src/test/java/org/apache/iceberg/hadoop/TestHadoopCatalog.java

+          .as("Table should have a spec with one void field")
+          .isEqualTo(v1Expected);
+    } else {
+      Assertions.assertThat(table.spec().isUnpartitioned())


This is actually a valid check on both versions now, unless we want to be sure to check for the void transform
https://github.com/apache/iceberg/blob/master/api/src/main/java/org/apache/iceberg/PartitionSpec.java#L89-L95

I feel like it is better to highlight the difference in this case.

RussellSpitzer · 2023-08-30T17:29:40Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java

+    } else {
+      expected =
+          PartitionSpecParser.fromJson(
+              table.schema(),


Not a blocker but I think we should expose whatever we need to do this programmatically to testing so we don't have to parse jsons.

Agreed, I'll create an issue to address that.

Filed #8434.

RussellSpitzer · 2023-08-30T18:17:42Z

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteDataFilesAction.java


  protected Table createTablePartitioned(int partitions, int files) {
-    return createTablePartitioned(partitions, files, SCALE, Maps.newHashMap());
+    Map<String, String> properties = ImmutableMap.of(TableProperties.FORMAT_VERSION, "1");


Why V1 For Rewrite DataFiles? (also just this particular create method?)

I can change that for a single test, actually. I'll update.

RussellSpitzer

Looks good to me, left a few minor questions. Some things to be done in followups

aokolnychyi · 2023-08-30T20:59:08Z

Thanks for reviewing everyone!

rdblue · 2023-09-02T20:41:04Z

core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java

    C catalog = catalog();

+    // TODO: temporarily ignore this test for REST catalogs (issue #8390)
+    Assumptions.assumeFalse(catalog instanceof RESTCatalog);


@aokolnychyi, while I understand wanting to get this in, I don't think it's a good idea to set a precedent that we will disable tests that aren't passing. I also don't see much description of the problem on #8390.

github-actions bot added the core label Aug 23, 2023

aokolnychyi added this to the Iceberg 1.4.0 milestone Aug 23, 2023

rdblue reviewed Aug 23, 2023

View reviewed changes

aokolnychyi force-pushed the v2-by-default branch from 90d8b60 to c3e613f Compare August 23, 2023 21:10

github-actions bot added spark hive labels Aug 23, 2023

aokolnychyi commented Aug 23, 2023

View reviewed changes

aokolnychyi force-pushed the v2-by-default branch 2 times, most recently from 7569eae to 8539085 Compare August 24, 2023 01:16

aokolnychyi mentioned this pull request Aug 24, 2023

Fix REST catalogs behavior with v2 tables #8390

Closed

aokolnychyi force-pushed the v2-by-default branch 2 times, most recently from 1500843 to c9b2be3 Compare August 25, 2023 15:30

nastra reviewed Aug 25, 2023

View reviewed changes

RussellSpitzer reviewed Aug 25, 2023

View reviewed changes

RussellSpitzer approved these changes Aug 25, 2023

View reviewed changes

aokolnychyi force-pushed the v2-by-default branch from c9b2be3 to aa47543 Compare August 28, 2023 23:44

aokolnychyi commented Aug 28, 2023

View reviewed changes

RussellSpitzer reviewed Aug 30, 2023

View reviewed changes

RussellSpitzer approved these changes Aug 30, 2023

View reviewed changes

Core: Use v2 format in new tables by default

825cc0b

aokolnychyi force-pushed the v2-by-default branch from aa47543 to 825cc0b Compare August 30, 2023 19:22

aokolnychyi mentioned this pull request Aug 30, 2023

Have a test utility method to programmatically create expected specs #8434

Closed

aokolnychyi merged commit 2f03e7a into apache:master Aug 30, 2023

rdblue reviewed Sep 2, 2023

View reviewed changes

rdblue mentioned this pull request Sep 2, 2023

Core: Fix REST catalog tests with default v2 tables #8469

Merged

wangtaohz mentioned this pull request Nov 26, 2023

[AMORO-2222] [Improvement]: Skip cleaning up dangling delete files for Iceberg V1 table apache/amoro#2361

Merged

3 tasks

zhongyujiang mentioned this pull request Dec 7, 2023

Docs: Update default format version to 2 #9239

Merged

nastra mentioned this pull request Jun 4, 2024

Core: Fix create v1 table on REST Catalog #10369

Merged

Core: Use V2 format in new tables by default #8381

Core: Use V2 format in new tables by default #8381

Uh oh!

Conversation

aokolnychyi commented Aug 23, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aokolnychyi commented Aug 23, 2023

Uh oh!

bryanck commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanck commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aokolnychyi commented Aug 24, 2023

Uh oh!

bryanck commented Aug 24, 2023

Uh oh!

aokolnychyi commented Aug 24, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aokolnychyi commented Aug 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RussellSpitzer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanck commented Aug 23, 2023 •

edited

Loading

bryanck commented Aug 23, 2023 •

edited

Loading

aokolnychyi commented Aug 25, 2023 •

edited

Loading