Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -76,4 +76,10 @@ default ComputeTableStats computeTableStats(Table table) {
throw new UnsupportedOperationException(
this.getClass().getName() + " does not implement computeTableStats");
}

/** Instantiates an action to rewrite all absolute paths in table metadata. */
default RewriteTablePath rewriteTablePath(Table table) {
throw new UnsupportedOperationException(
this.getClass().getName() + " does not implement rewriteTablePath");
}
}
103 changes: 103 additions & 0 deletions api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iceberg.actions;

/**
* An action that rewrites the table's metadata files to a staging directory, replacing all source
* prefixes in absolute paths with a specified target prefix. There are two modes:
*
* <ul>
* <li><b>Complete copy:</b> Rewrites all metadata files to the staging directory.
* <li><b>Incremental copy:</b> Rewrites a subset of metadata files to the staging directory,
* consisting of metadata files added since a specified start version and/or until end
* version. The start/end version is identified by the name of a metadata.json file, and all
* metadata files added before/after these file are marked for rewrite.
* </ul>
*
* This action can be used as the starting point to fully or incrementally copy an Iceberg table
* located under the source prefix to the target prefix.
*
* <p>The action returns the following:
*
* <ol>
* <li>The name of the latest metadata.json rewritten to staging location. After the files are
* copied, this will be the root of the copied table.
* <li>A list of all files added to the table between startVersion and endVersion, including their
* original and target paths under the target prefix. This list covers both original and
* rewritten files, allowing for copying to the target paths to form the copied table.
* </ol>
*/
public interface RewriteTablePath extends Action<RewriteTablePath, RewriteTablePath.Result> {
Copy link
Member

@szehon-ho szehon-ho Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a overall javadoc describing this. In particular, I think the concept of 'target version' needs to be explained here. I think its clearer to say 'last metadata version'

Rewrites a set of the table's metadata files to a staging directory, replacing all source prefixes in absolute paths with a specified target prefix.

There are two modes.  
1. A complete copy rewrites all metadata files to the staging directory.
2. An incremental copy rewrites a set of metadata files to the staging directory, consisting of metadata files added since a specified 'last metadata version'.

For incremental copy, the 'last metadata version' is specified by a name of a metadata.json file, with all metadata files added after this file being marked for rewrite.

The action can be used as the starting point to fully or incrementally copy an Iceberg table located under the source prefix to the target prefix. It returns the following:
1. The name of the latest metadata.json file copied.
2. A listing of rewritten table metadata files, relative to staging directory, that can be copied to the target prefix.
3. A listing of table data files, relative to source prefix, that can be copied to the target prefix.

@flyrain does it make sense?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thanks for the suggestion.

Copy link
Member

@szehon-ho szehon-ho Oct 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

To update with our latest conversation, how about replace the last list with


1. A listing of all files added in the table between startVersion and endVersion, with both the original path and a target path under the target prefix.  This list includes files in the original table and those that have been rewritten to staging.  All files can be copied to their specified target path to form the copied table.
2. The name of the latest metadata.json rewritten to staging.  After the files are copy, this can become the root of the copied table.


/**
* Configure a source prefix that will be replaced by the specified target prefix in all paths
*
* @param sourcePrefix the source prefix to be replaced
* @param targetPrefix the target prefix
* @return this for method chaining
*/
RewriteTablePath rewriteLocationPrefix(String sourcePrefix, String targetPrefix);
Copy link
Member

@szehon-ho szehon-ho Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more thing came up while discussing internally. This was brought up by @huaxiangsun today. Its possible that the data and metadata are in different locations. In this case, we would need an optional API

rewriteLocationPrefix(String sourceDataPrefix, String sourceMetadataPrefix, String targetDataPrefix, String targetMetadataPrefix)

What do you think @laithalzyoud @flyrain let me know if we missed something though.

Copy link
Contributor

@flyrain flyrain Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's useful, and I'm OK with either adding it in this PR or in a followup PR. Another option is to add a list of mappings like this {(sourcePrefix1, targetPrefix1), (sourcePrefix2, targetPrefix2), ...}. We will need to figure out the overlap though, but I think it is solvable. It's a not blocker for me.

Copy link
Member

@szehon-ho szehon-ho Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a use case for something beyond data and metadata? Just wondering if that's unnecessary complicated. With this four-arg API we can just check data/metadata dont overlap, else throw exception and mention they can use the two-arg API?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Theoretically, data/metadata file could be anywhere. For example, there is a use case where people store old partitions in a cheaper storage. I understand that's not a common use case, but just like the different directories between data files and metadata files, it could happen.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean. Makes sense, we can tackle that in follow on then as you mentioned.


/**
* First metadata version to rewrite, identified by name of a metadata.json file in the table's
* metadata log. It is optional, if provided then this action will only rewrite metadata files
* added after this version.
*
* @param startVersion name of a metadata.json file. For example,
* "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json".
* @return this for method chaining
*/
RewriteTablePath startVersion(String startVersion);

/**
* Last metadata version to rewrite, identified by name of a metadata.json file in the table's
* metadata log. It is optional, if provided then this action will only rewrite metadata files
* added before this file, including the file itself.
*
* @param endVersion name of a metadata.json file. For example,
* "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json".
* @return this for method chaining
*/
RewriteTablePath endVersion(String endVersion);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

endVersion is not clear. What do you think about copyVersion?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use copyVersion if only a single version will be rewritten by the script, however the original implementation included functionality to rewrite all version between lastCopiedVersion and endVersion

Copy link
Member

@szehon-ho szehon-ho Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about:

Last metadata version to copy, identified by name of a metadata.json file in the table's metadata log. This action will rewrite all metadata files added before this, up to this file itself.

Can we not support HadoopFileIO initially, so we can simplify definition of the argument?


/**
* Custom staging location. It is optional. By default, staging location is a subdirectory under
* table's metadata directory.
*
* @param stagingLocation the staging location
* @return this for method chaining
*/
RewriteTablePath stagingLocation(String stagingLocation);

/** The action result that contains a summary of the execution. */
interface Result {
Copy link
Member

@szehon-ho szehon-ho Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually a few more comments here, I think we need additional list for

  • position delete files
  • equality delete files
  • puffin files
    Does it make sense?

Also, what do you guys think to return source, target path pair for each file. Each source path is either the original file path in the table, or the rewritten one in staging. The target path is the one under target prefix (the target one is always the latest referred to by all rewritten metadata paths). This would make the subsequent move is easier to perform. I know in the earlier implementation this existed.

cc @flyrain for thoughts

Copy link
Contributor Author

@laithalzyoud laithalzyoud Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@szehon-ho I don't think it makes sense, because metadataFileList file will contain all metadata files paths (metadata.json, manifests, manifest lists, position/equality delete files etc), so there's no point of having multiple lists. I agree with returning source and target path for each file, will keep that in mind for the implementation PR.

Copy link
Member

@szehon-ho szehon-ho Oct 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea i thought a bit over the weekend, and multiple list is too complicated.

How about we dont need multiple list (metadataFileList and dataFileList), and just have one list (fileList). My main concern is deleteFile, which list it should be part of. It doesnt truly fit into either. I feel In the end, we want to have just a list of Pair<String, String> , which is source and target. Source can be source if rewritten, or original source if not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since deleteFiles already exists under the metadata folder in the storage then it makes sense to have them in the metadataFileList. I think the expectation was for the user to copy all files in metadataFileList to /metadata and dataFileList to /data in the new target location. But I agree with your suggestion, we can further narrow it down to one list as well and just have the source,target paths for all of them and the user can just initiate the copy from source to target without worrying about if it's data or metadata files 👍

Copy link
Member

@szehon-ho szehon-ho Oct 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since deleteFiles already exists under the metadata folder in the storage

I thought its generally not the case, and they co-exist with the data files in the partition folder. Hence my initial comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DeleteFiles are not considered as metadata files even they are in the directory /metadata. It's more and more clear to me that we may just use one list holding pairs of (sourcePath, targetPath). WDYT?

/** Staging location of rewritten files */
String stagingLocation();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: let's capitalize these field javadocs?


/**
* Path to a comma-separated list of source and target paths for all files added to the table
* between startVersion and endVersion, including original data files and metadata files
* rewritten to staging.
*/
String fileListLocation();

/** Name of latest metadata file version */
String latestVersion();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iceberg.actions;

import org.immutables.value.Value;

@Value.Enclosing
@SuppressWarnings("ImmutablesStyle")
@Value.Style(
typeImmutableEnclosing = "ImmutableRewriteTablePath",
visibilityString = "PUBLIC",
builderVisibilityString = "PUBLIC")
interface BaseRewriteTablePath extends RewriteTablePath {

@Value.Immutable
interface Result extends RewriteTablePath.Result {}
}