Spark: Add RewriteTablePath action interface #10920

laithalzyoud · 2024-08-12T14:35:42Z

A follow up PR on #10024

laithalzyoud · 2024-08-12T14:41:46Z

Does everyone agree that CopyTable is a good name for this action? Since it's just rewriting metadata, manifest and position delete files with the new location prefix and not actually copying the table, maybe it's better to name it something more specific like RewriteTableLocation or something similar

manuzhang · 2024-08-12T14:58:09Z

Since it's just rewriting metadata, manifest and position delete files with the new location prefix and not actually copying the table

Isn't an implementation of this interface actually copying the table?

laithalzyoud · 2024-08-12T15:43:06Z

Since it's just rewriting metadata, manifest and position delete files with the new location prefix and not actually copying the table

Isn't an implementation of this interface actually copying the table?

There is an explanation in the original PR that explains the functionality of this action

flyrain · 2024-08-12T17:40:25Z

cc @szehon-ho @huaxingao

flyrain · 2024-08-12T17:45:19Z

Isn't an implementation of this interface actually copying the table?

NO. I think it makes sense to go with a name like RewriteFileLocation, or RewriteFilePaths

manuzhang · 2024-08-13T03:44:39Z

I'm a bit lost in the context. If I run this action without copying the table (e.g. rewrite s3 to gcs), isn't it in a corrupt status?

laithalzyoud · 2024-08-13T14:09:49Z

I'm a bit lost in the context. If I run this action without copying the table (e.g. rewrite s3 to gcs), isn't it in a corrupt status?

@manuzhang Not really, since the rewritten files are stored in a staging directory, when you copy the new table to the new location you copy all the data files + the rewritten metadata files from the staging location and ignore the old metadata files. So the table in the old location will not be corrupted 👍

api/src/main/java/org/apache/iceberg/actions/ActionsProvider.java

flyrain · 2024-08-13T18:29:30Z

api/src/main/java/org/apache/iceberg/actions/RewriteTableLocation.java

+
+import org.apache.iceberg.Table;
+
+public interface RewriteTableLocation extends Action<RewriteTableLocation, RewriteTableLocation.Result> {


I'd try to avoid using the name TableLocation, which is a dedicated name for the location of the table as a table property. Given the action is to rewrite the file paths within all metadata files, is rewriteFilePath more suitable?

To be more specific, it's rewriteMetadataFilePath?

Mostly in metadata files. I think it has to touch position/eq delete files as well as they have the absolute file paths as well.

@flyrain Yes it does modify position files as well. RewriteFilePath implies that the method rewrites a single file. Maybe RewriteTablePath?

Yeah, the path rewritten could be a directory as well. I think RewriteTablePath is fine, other alternative cuold be RewriteAbsolutePath or Just RewritePath.

flyrain · 2024-08-21T19:04:06Z

@amogh-jahagirdar @nastra, @anuragmantri, you may be interested as well.

anuragmantri · 2024-08-21T19:38:45Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   *     "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json".
+   * @return this for method chaining
+   */
+  RewriteTablePath endVersion(String endVersion);


endVersion is not clear. What do you think about copyVersion?

We can use copyVersion if only a single version will be rewritten by the script, however the original implementation included functionality to rewrite all version between lastCopiedVersion and endVersion

How about:

Last metadata version to copy, identified by name of a metadata.json file in the table's metadata log. This action will rewrite all metadata files added before this, up to this file itself.

Can we not support HadoopFileIO initially, so we can simplify definition of the argument?

anuragmantri · 2024-08-21T19:40:30Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+  RewriteTablePath stagingLocation(String stagingLocation);
+
+  /**
+   * Set the target table. It is optional if the start version is provided.


What is start version here? Could you clarify where start version is provided?

Start version is the same as lastCopiedVersion, maybe we should name them as startVersion and endVersion, or remove this and always rewrite the paths for the full table. @flyrain Do you remember why you added such a feature?

anuragmantri · 2024-08-21T19:44:22Z

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePathActionResult.java

+  }
+
+  @Override
+  public String dataFileListLocation() {


Nit: Should we clarify more by renaming them targetDataLocation, targetMetadataLocation and copiedVersion ?

These are the location for a .txt list of files (data/metadata) to copy, so I don't think the suggested names are suitable for them

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePathActionResult.java

…into add-copy-table-action-interface # Conflicts: # api/src/main/java/org/apache/iceberg/actions/ActionsProvider.java

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePath.java

…Path.java Fix typo Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

laithalzyoud · 2024-09-19T12:53:55Z

Hello @nastra @anuragmantri! What's left to merge this PR?

… into add-copy-table-action-interface

laithalzyoud · 2024-09-24T20:36:28Z

@nastra can you rerun the workflows please? I fixed a formatting error

nastra · 2024-09-25T09:05:56Z

@nastra can you rerun the workflows please? I fixed a formatting error

The formatting error seems to still exist

laithalzyoud · 2024-09-25T10:00:33Z

@nastra can you rerun the workflows please? I fixed a formatting error

The formatting error seems to still exist

Should be ok now. Can you rerun again?

laithalzyoud · 2024-09-27T15:45:29Z

Hey @nastra @manuzhang @anuragmantri @flyrain! Can we merge this now?

flyrain

Thanks @laithalzyoud for working on it. LGTM. Left one minor question.

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePath.java

szehon-ho

Hi, @laithalzyoud and @flyrain . I think the Action is valuable but a bit confusing at first glance to user. I took a look and left some comments to try to make it cleaner.

szehon-ho · 2024-10-01T21:20:05Z

api/src/main/java/org/apache/iceberg/actions/ActionsProvider.java

        this.getClass().getName() + " does not implement computeTableStats");
  }
+
+  /** Instantiates an action to rewrite table files absolute path. */


It does not seem like a correct sentence, what do you think of: 'rewrite all absolute paths in table metadata'?

szehon-ho · 2024-10-01T21:21:59Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+public interface RewriteTablePath extends Action<RewriteTablePath, RewriteTablePath.Result> {
+
+  /**
+   * Passes the source and target prefixes that will be used to replace the source prefix with the


The sentence repeats itself, what do you think of 'Configure a source prefix that will be replaced by the specified target prefix in all paths'

szehon-ho · 2024-10-01T21:24:01Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+
+import org.apache.iceberg.Table;
+
+public interface RewriteTablePath extends Action<RewriteTablePath, RewriteTablePath.Result> {


I think we need a overall javadoc describing this. In particular, I think the concept of 'target version' needs to be explained here. I think its clearer to say 'last metadata version'

Rewrites a set of the table's metadata files to a staging directory, replacing all source prefixes in absolute paths with a specified target prefix. There are two modes. 1. A complete copy rewrites all metadata files to the staging directory. 2. An incremental copy rewrites a set of metadata files to the staging directory, consisting of metadata files added since a specified 'last metadata version'. For incremental copy, the 'last metadata version' is specified by a name of a metadata.json file, with all metadata files added after this file being marked for rewrite. The action can be used as the starting point to fully or incrementally copy an Iceberg table located under the source prefix to the target prefix. It returns the following: 1. The name of the latest metadata.json file copied. 2. A listing of rewritten table metadata files, relative to staging directory, that can be copied to the target prefix. 3. A listing of table data files, relative to source prefix, that can be copied to the target prefix.

@flyrain does it make sense?

+1, thanks for the suggestion.

Thanks.

To update with our latest conversation, how about replace the last list with

1. A listing of all files added in the table between startVersion and endVersion, with both the original path and a target path under the target prefix. This list includes files in the original table and those that have been rewritten to staging. All files can be copied to their specified target path to form the copied table. 2. The name of the latest metadata.json rewritten to staging. After the files are copy, this can become the root of the copied table.

szehon-ho · 2024-10-01T22:32:53Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   *     "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json".
+   * @return this for method chaining
+   */
+  RewriteTablePath lastCopiedVersion(String lastCopiedVersion);


Based on previous suggestion, how about
Last copied metadata version, identified by name of a metadata.json file in the table's metadata log. This action will rewrite all metadata files added after this.

Can we not support HadoopFileIO initially, so we can simplify definition of the argument?

szehon-ho · 2024-10-01T22:34:44Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   *     "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json".
+   * @return this for method chaining
+   */
+  RewriteTablePath endVersion(String endVersion);


How about:

Last metadata version to copy, identified by name of a metadata.json file in the table's metadata log. This action will rewrite all metadata files added before this, up to this file itself.

Can we not support HadoopFileIO initially, so we can simplify definition of the argument?

szehon-ho · 2024-10-01T23:24:40Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+
+  /** The action result that contains a summary of the execution. */
+  interface Result {
+    /** Return staging location */


Nit: omit 'return' as its redundant.

'staging location of rewritten files'

szehon-ho · 2024-10-01T23:26:48Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+    String stagingLocation();
+
+    /** Return directory of data files list. */
+    String dataFileListLocation();


This is a directory, or is it a path to the listing? I thought it was path according to https://github.com/apache/iceberg/pull/10024/files

It's a directory actually, but I think we can make it a file path in the implementation for a better UX

szehon-ho · 2024-10-01T23:41:35Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   * @param targetTable the target table
+   * @return this for method chaining
+   */
+  RewriteTablePath targetTable(Table targetTable);


This is just a convenience to get the last metadata.json file location , right? That table is never used elsewhere here.

What do you guys think to omit this one (I feel the user can load the table independently and get the path). As it is now, its pretty confusing in my opinion.

Yes I also think it's unnecessary, startVersion should be enough

The original intent was to provide flexibility in case Spark could access both tables. However, I now agree with your point—it makes sense to keep the interface simpler by removing that part.

szehon-ho · 2024-10-04T22:31:44Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+  RewriteTablePath targetTable(Table targetTable);
+
+  /** The action result that contains a summary of the execution. */
+  interface Result {


Actually a few more comments here, I think we need additional list for

position delete files

equality delete files

puffin files
Does it make sense?

Also, what do you guys think to return source, target path pair for each file. Each source path is either the original file path in the table, or the rewritten one in staging. The target path is the one under target prefix (the target one is always the latest referred to by all rewritten metadata paths). This would make the subsequent move is easier to perform. I know in the earlier implementation this existed.

cc @flyrain for thoughts

@szehon-ho I don't think it makes sense, because metadataFileList file will contain all metadata files paths (metadata.json, manifests, manifest lists, position/equality delete files etc), so there's no point of having multiple lists. I agree with returning source and target path for each file, will keep that in mind for the implementation PR.

yea i thought a bit over the weekend, and multiple list is too complicated.

How about we dont need multiple list (metadataFileList and dataFileList), and just have one list (fileList). My main concern is deleteFile, which list it should be part of. It doesnt truly fit into either. I feel In the end, we want to have just a list of Pair<String, String> , which is source and target. Source can be source if rewritten, or original source if not.

Since deleteFiles already exists under the metadata folder in the storage then it makes sense to have them in the metadataFileList. I think the expectation was for the user to copy all files in metadataFileList to /metadata and dataFileList to /data in the new target location. But I agree with your suggestion, we can further narrow it down to one list as well and just have the source,target paths for all of them and the user can just initiate the copy from source to target without worrying about if it's data or metadata files 👍

Since deleteFiles already exists under the metadata folder in the storage

I thought its generally not the case, and they co-exist with the data files in the partition folder. Hence my initial comment.

DeleteFiles are not considered as metadata files even they are in the directory /metadata. It's more and more clear to me that we may just use one list holding pairs of (sourcePath, targetPath). WDYT?

laithalzyoud · 2024-10-07T16:01:19Z

@szehon-ho I addressed the comments! PTAL 👍

laithalzyoud · 2024-10-10T11:44:55Z

@szehon-ho PTAL

szehon-ho · 2024-10-14T17:14:36Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+    String stagingLocation();
+
+    /**
+     * path to a list of comma-separated pairs of source and target path for data and metadata


How about:

Path to a list of comma-separated pairs of source path to target path for all files in the table added between startVersion and endVersion. Note that this list includes data files from the original table and metadata files rewritten to staging.

Also tagging @KolonelKustard who is planning to consume this functionality if any thoughts on whether this is nice to consume for the copy part.

+1

Having a single list means that as the table format evolves and the implementations of this copy action evolve, hopefully there's fewer changes for downstream users of this action.

Having the target path saves downstream users having to duplicate the prefix re-writing rules already applied by this actions implementations. Which hopefully makes using this a bit less error-prone.

szehon-ho

Sorry for delay. Looking better, I added some more comments.

szehon-ho · 2024-10-14T17:16:20Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+  /** The action result that contains a summary of the execution. */
+  interface Result {
+    /** staging location of rewritten files */
+    String stagingLocation();


Nit: let's capitalize these field javadocs?

szehon-ho · 2024-10-14T17:38:33Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+
+import org.apache.iceberg.Table;
+
+public interface RewriteTablePath extends Action<RewriteTablePath, RewriteTablePath.Result> {


Thanks.

To update with our latest conversation, how about replace the last list with

1. A listing of all files added in the table between startVersion and endVersion, with both the original path and a target path under the target prefix. This list includes files in the original table and those that have been rewritten to staging. All files can be copied to their specified target path to form the copied table. 2. The name of the latest metadata.json rewritten to staging. After the files are copy, this can become the root of the copied table.

laithalzyoud · 2024-10-16T12:26:48Z

Thanks for the review @szehon-ho. I addressed the comments, PTAL

szehon-ho

Thanks it looks good to me. Just one more thing came up for me, what do you think?

szehon-ho · 2024-10-16T20:55:12Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   * @param targetPrefix the target prefix
+   * @return this for method chaining
+   */
+  RewriteTablePath rewriteLocationPrefix(String sourcePrefix, String targetPrefix);


One more thing came up while discussing internally. This was brought up by @huaxiangsun today. Its possible that the data and metadata are in different locations. In this case, we would need an optional API

rewriteLocationPrefix(String sourceDataPrefix, String sourceMetadataPrefix, String targetDataPrefix, String targetMetadataPrefix)

What do you think @laithalzyoud @flyrain let me know if we missed something though.

I think it's useful, and I'm OK with either adding it in this PR or in a followup PR. Another option is to add a list of mappings like this {(sourcePrefix1, targetPrefix1), (sourcePrefix2, targetPrefix2), ...}. We will need to figure out the overlap though, but I think it is solvable. It's a not blocker for me.

Is there a use case for something beyond data and metadata? Just wondering if that's unnecessary complicated. With this four-arg API we can just check data/metadata dont overlap, else throw exception and mention they can use the two-arg API?

Theoretically, data/metadata file could be anywhere. For example, there is a use case where people store old partitions in a cheaper storage. I understand that's not a common use case, but just like the different directories between data files and metadata files, it could happen.

I see what you mean. Makes sense, we can tackle that in follow on then as you mentioned.

szehon-ho · 2024-10-17T21:29:49Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   * @param targetPrefix the target prefix
+   * @return this for method chaining
+   */
+  RewriteTablePath rewriteLocationPrefix(String sourcePrefix, String targetPrefix);


I see what you mean. Makes sense, we can tackle that in follow on then as you mentioned.

szehon-ho · 2024-10-18T18:01:54Z

Merged , thanks @laithalzyoud

laithalzyoud · 2024-10-18T18:09:38Z

Thanks @szehon-ho! Will follow up with the implementation PR soon 👍

flyrain · 2024-10-18T18:40:07Z

Great! Thanks a lot for working on it, guys!

Add CopyTable action interface

71afb73

github-actions bot added API core labels Aug 12, 2024

laithalzyoud changed the title ~~Add CopyTable action interface~~ Spark: Add CopyTable action interface Aug 12, 2024

laithalzyoud mentioned this pull request Aug 12, 2024

Spark: Add CopyTable spark action #10024

Closed

Change name to RewriteTableLocation

1c9a028

laithalzyoud changed the title ~~Spark: Add CopyTable action interface~~ Spark: Add RewriteTableLocaiton action interface Aug 13, 2024

laithalzyoud changed the title ~~Spark: Add RewriteTableLocaiton action interface~~ Spark: Add RewriteTableLocation action interface Aug 13, 2024

flyrain reviewed Aug 13, 2024

View reviewed changes

Change name to RewriteTablePath

9575985

laithalzyoud changed the title ~~Spark: Add RewriteTableLocation action interface~~ Spark: Add RewriteTablePath action interface Aug 18, 2024

Rename copyTable to rewriteTablePath

e8e59d0

anuragmantri reviewed Aug 21, 2024

View reviewed changes

nastra reviewed Sep 11, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePathActionResult.java Outdated Show resolved Hide resolved

Laith Alzyoud added 2 commits September 16, 2024 16:56

Use Immutable for RewriteTablePath

7755aff

Merge branch 'main' of https://github.com/revolut-engineering/iceberg …

d750517

…into add-copy-table-action-interface # Conflicts: # api/src/main/java/org/apache/iceberg/actions/ActionsProvider.java

nastra reviewed Sep 17, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePath.java Outdated Show resolved Hide resolved

Update core/src/main/java/org/apache/iceberg/actions/BaseRewriteTable…

05d904e

…Path.java Fix typo Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com>

Laith Alzyoud added 2 commits September 24, 2024 11:25

Fix formatting

dc6b9a3

Merge remote-tracking branch 'origin/add-copy-table-action-interface'…

7205a6c

… into add-copy-table-action-interface

Fix formatting again

154e8dd

flyrain approved these changes Oct 1, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/actions/BaseRewriteTablePath.java Show resolved Hide resolved

szehon-ho reviewed Oct 1, 2024

View reviewed changes

szehon-ho reviewed Oct 4, 2024

View reviewed changes

Address comments

7039f2b

Return only one list of files

3c72599

szehon-ho reviewed Oct 14, 2024

View reviewed changes

Update javadocs

9addc9c

szehon-ho reviewed Oct 16, 2024

View reviewed changes

szehon-ho approved these changes Oct 17, 2024

View reviewed changes

szehon-ho merged commit d61a98d into apache:main Oct 18, 2024

szehon-ho mentioned this pull request Nov 15, 2024

Spark 3.5: Implement RewriteTablePath #11555

Merged

zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request Dec 23, 2024

API: Add RewriteTablePath action interface (apache#10920)

7945cc1


		import org.apache.iceberg.Table;

		public interface RewriteTableLocation extends Action<RewriteTableLocation, RewriteTableLocation.Result> {


		import org.apache.iceberg.Table;

		public interface RewriteTablePath extends Action<RewriteTablePath, RewriteTablePath.Result> {

Spark: Add RewriteTablePath action interface #10920

Spark: Add RewriteTablePath action interface #10920

Conversation

laithalzyoud commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

laithalzyoud commented Aug 12, 2024

Uh oh!

manuzhang commented Aug 12, 2024

Uh oh!

laithalzyoud commented Aug 12, 2024

Uh oh!

flyrain commented Aug 12, 2024

Uh oh!

flyrain commented Aug 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

manuzhang commented Aug 13, 2024

Uh oh!

laithalzyoud commented Aug 13, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flyrain commented Aug 21, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szehon-ho Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

laithalzyoud Aug 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

laithalzyoud commented Sep 19, 2024

Uh oh!

laithalzyoud commented Sep 24, 2024

Uh oh!

nastra commented Sep 25, 2024

Uh oh!

laithalzyoud commented Sep 25, 2024

Uh oh!

laithalzyoud commented Sep 27, 2024

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szehon-ho Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

laithalzyoud commented Aug 12, 2024 •

edited

Loading

flyrain commented Aug 12, 2024 •

edited

Loading

szehon-ho Oct 1, 2024 •

edited

Loading

laithalzyoud Aug 30, 2024 •

edited

Loading

szehon-ho Oct 1, 2024 •

edited

Loading

szehon-ho Oct 14, 2024 •

edited

Loading

szehon-ho Oct 1, 2024 •

edited

Loading

szehon-ho Oct 1, 2024 •

edited

Loading

szehon-ho Oct 1, 2024 •

edited

Loading

szehon-ho Oct 4, 2024 •

edited

Loading

laithalzyoud Oct 7, 2024 •

edited

Loading

szehon-ho Oct 7, 2024 •

edited

Loading

szehon-ho Oct 14, 2024 •

edited

Loading