Spark: Add procedure to publish WAP changes using wap.id #4715
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
PTAL @rdblue when you have a chance. Thanks! |
Contributor
|
@edgarRd, sorry I missed this. Can you rebase and I'll review? |
a57181b to
1273c85
Compare
Contributor
Author
|
Thanks @rdblue - I've rebased the branch. |
Contributor
|
@RussellSpitzer, can you also take a look at this? |
rdblue
approved these changes
Jul 6, 2022
Contributor
|
Thanks, @edgarRd! Could you also port this to Spark 3.3? |
Contributor
Author
|
Thank you, @rdblue - I'll send a PR for porting to Spark 3.3 in a bit. |
Contributor
|
Thanks @edgarRd !! should we also add this procedure in Spark Procedures Doc |
Contributor
Author
Yeah, good catch! I can add the docs in the follow up PR I have for adding it to Spark 3.3: #5223 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While the WAP workflow enables users to write data via SQL using a
wap_id, there's a lack in usability when applying those changes written to the table during the Publish step of the workflow. The user needs to figure out thesnapshot-idprogrammatically to cherry-pick the changes. Ideally, we should have all steps of the WAP workflow available via SQL, since Iceberg has thewap-id => snapshot-idmapping in its own metadata.This PR proposes a SQL procedure to cherry-pick the changes created with a
wap-id. Functionally, it works the same as the cherry-pick procedure, but receives awap-idas argument instead of asnapshot-id. This would make the Write and Publish parts of WAP available via SQL. The procedure name proposed ispublish_changesbut I'm open to suggestions if another name would fit better.Thanks.
BTW - I considered extending the current
cherry-pickprocedure but I figured the implementation wouldn't look that much different nor we'd end up with less code since there's a few things that'd need to be overwritten from an already simple procedure; so mostly it would've been coupling both implementations. Open to suggestions.