-
Notifications
You must be signed in to change notification settings - Fork 3k
Spark: Add RewriteTablePath action interface #10920
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
71afb73
1c9a028
9575985
e8e59d0
7755aff
d750517
05d904e
dc6b9a3
7205a6c
154e8dd
7039f2b
3c72599
9addc9c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.iceberg.actions; | ||
|
|
||
| /** | ||
| * An action that rewrites the table's metadata files to a staging directory, replacing all source | ||
| * prefixes in absolute paths with a specified target prefix. There are two modes: | ||
| * | ||
| * <ul> | ||
| * <li><b>Complete copy:</b> Rewrites all metadata files to the staging directory. | ||
| * <li><b>Incremental copy:</b> Rewrites a subset of metadata files to the staging directory, | ||
| * consisting of metadata files added since a specified start version and/or until end | ||
| * version. The start/end version is identified by the name of a metadata.json file, and all | ||
| * metadata files added before/after these file are marked for rewrite. | ||
| * </ul> | ||
| * | ||
| * This action can be used as the starting point to fully or incrementally copy an Iceberg table | ||
| * located under the source prefix to the target prefix. | ||
| * | ||
| * <p>The action returns the following: | ||
| * | ||
| * <ol> | ||
| * <li>The name of the latest metadata.json rewritten to staging location. After the files are | ||
| * copied, this will be the root of the copied table. | ||
| * <li>A list of all files added to the table between startVersion and endVersion, including their | ||
| * original and target paths under the target prefix. This list covers both original and | ||
| * rewritten files, allowing for copying to the target paths to form the copied table. | ||
| * </ol> | ||
| */ | ||
| public interface RewriteTablePath extends Action<RewriteTablePath, RewriteTablePath.Result> { | ||
|
|
||
| /** | ||
| * Configure a source prefix that will be replaced by the specified target prefix in all paths | ||
| * | ||
| * @param sourcePrefix the source prefix to be replaced | ||
| * @param targetPrefix the target prefix | ||
| * @return this for method chaining | ||
| */ | ||
| RewriteTablePath rewriteLocationPrefix(String sourcePrefix, String targetPrefix); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One more thing came up while discussing internally. This was brought up by @huaxiangsun today. Its possible that the data and metadata are in different locations. In this case, we would need an optional API What do you think @laithalzyoud @flyrain let me know if we missed something though.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's useful, and I'm OK with either adding it in this PR or in a followup PR. Another option is to add a list of mappings like this
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a use case for something beyond data and metadata? Just wondering if that's unnecessary complicated. With this four-arg API we can just check data/metadata dont overlap, else throw exception and mention they can use the two-arg API?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Theoretically, data/metadata file could be anywhere. For example, there is a use case where people store old partitions in a cheaper storage. I understand that's not a common use case, but just like the different directories between data files and metadata files, it could happen.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see what you mean. Makes sense, we can tackle that in follow on then as you mentioned. |
||
|
|
||
| /** | ||
| * First metadata version to rewrite, identified by name of a metadata.json file in the table's | ||
| * metadata log. It is optional, if provided then this action will only rewrite metadata files | ||
| * added after this version. | ||
| * | ||
| * @param startVersion name of a metadata.json file. For example, | ||
| * "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json". | ||
| * @return this for method chaining | ||
| */ | ||
| RewriteTablePath startVersion(String startVersion); | ||
|
|
||
| /** | ||
| * Last metadata version to rewrite, identified by name of a metadata.json file in the table's | ||
| * metadata log. It is optional, if provided then this action will only rewrite metadata files | ||
| * added before this file, including the file itself. | ||
| * | ||
| * @param endVersion name of a metadata.json file. For example, | ||
| * "00001-8893aa9e-f92e-4443-80e7-cfa42238a654.metadata.json". | ||
| * @return this for method chaining | ||
| */ | ||
| RewriteTablePath endVersion(String endVersion); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can use
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How about:
Can we not support HadoopFileIO initially, so we can simplify definition of the argument? |
||
|
|
||
| /** | ||
| * Custom staging location. It is optional. By default, staging location is a subdirectory under | ||
| * table's metadata directory. | ||
| * | ||
| * @param stagingLocation the staging location | ||
| * @return this for method chaining | ||
| */ | ||
| RewriteTablePath stagingLocation(String stagingLocation); | ||
|
|
||
| /** The action result that contains a summary of the execution. */ | ||
| interface Result { | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually a few more comments here, I think we need additional list for
Also, what do you guys think to return source, target path pair for each file. Each source path is either the original file path in the table, or the rewritten one in staging. The target path is the one under target prefix (the target one is always the latest referred to by all rewritten metadata paths). This would make the subsequent move is easier to perform. I know in the earlier implementation this existed. cc @flyrain for thoughts
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @szehon-ho I don't think it makes sense, because
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yea i thought a bit over the weekend, and multiple list is too complicated. How about we dont need multiple list (metadataFileList and dataFileList), and just have one list (fileList). My main concern is deleteFile, which list it should be part of. It doesnt truly fit into either. I feel In the end, we want to have just a list of Pair<String, String> , which is source and target. Source can be source if rewritten, or original source if not.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I thought its generally not the case, and they co-exist with the data files in the partition folder. Hence my initial comment.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. DeleteFiles are not considered as metadata files even they are in the directory |
||
| /** Staging location of rewritten files */ | ||
| String stagingLocation(); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: let's capitalize these field javadocs? |
||
|
|
||
| /** | ||
| * Path to a comma-separated list of source and target paths for all files added to the table | ||
| * between startVersion and endVersion, including original data files and metadata files | ||
| * rewritten to staging. | ||
| */ | ||
| String fileListLocation(); | ||
|
|
||
| /** Name of latest metadata file version */ | ||
| String latestVersion(); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
| package org.apache.iceberg.actions; | ||
|
|
||
| import org.immutables.value.Value; | ||
|
|
||
| @Value.Enclosing | ||
| @SuppressWarnings("ImmutablesStyle") | ||
| @Value.Style( | ||
| typeImmutableEnclosing = "ImmutableRewriteTablePath", | ||
| visibilityString = "PUBLIC", | ||
| builderVisibilityString = "PUBLIC") | ||
| interface BaseRewriteTablePath extends RewriteTablePath { | ||
flyrain marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| @Value.Immutable | ||
| interface Result extends RewriteTablePath.Result {} | ||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need a overall javadoc describing this. In particular, I think the concept of 'target version' needs to be explained here. I think its clearer to say 'last metadata version'
@flyrain does it make sense?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, thanks for the suggestion.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
To update with our latest conversation, how about replace the last list with