Hive: Set commit state as Unknown before throwing CommitStateUnknownException #7931
Conversation
|
cc @pvary and @szehon-ho, please help to review this when you are free. |
| ops.refresh(); | ||
|
|
||
| Assertions.assertThat(ops.current()) | ||
| .as("Current metadata should have changed") |
There was a problem hiding this comment.
Nit: "Current metadata should have changed" -> should not have changed
There was a problem hiding this comment.
To make this clearer can we instead check that the metadata is the correct new snapshot? We should be able to get this in our mock.
There was a problem hiding this comment.
@RussellSpitzer, updated to check whether the metadata location and the schema are correct.
There was a problem hiding this comment.
Thanks, @RussellSpitzer for the good suggestion. And I just verified that the TableMetadata has not customized the equals. So there are some other UTs should be improved. Should we do that in another patch or combine it in this one?
There was a problem hiding this comment.
Hmm, for now could we just check locations? Then we can do a full equal implementation in another PR
|
Thanks for the fix @ConeyLiu! Left one small comment for the test |
| .when(spyOps) | ||
| .persistTable(any(), anyBoolean(), any()); | ||
|
|
||
| Assertions.assertThatThrownBy(() -> spyOps.commit(metadataV2, metadataV1)) |
There was a problem hiding this comment.
Hi @pvary, this UT is adapted from the above UT and here we commit the metadataV2 back to metadataV1. And we throw the LockException after the commit succeeded. So the following should be Current metadata should have changed.
RussellSpitzer
left a comment
There was a problem hiding this comment.
LGTM, do we need to backport this and do a 1.3.1 release?
I have one suggestion about the test, in cases like this where we are checking for success it's best to check that we have the new expected snapshot value and not just that we do not have the old value.
|
Thanks @pvary @szehon-ho @RussellSpitzer for the reviewing.
It would be better to do that. Should I submit a PR to the 1.3.x branch? I can't find it. |
|
Thanks @ConeyLiu for the PR and @szehon-ho and @RussellSpitzer for the review! |
|
@ConeyLiu: The 1.3.x branch is missing. Maybe we forgot create it during the release process. |
|
Thanks @pvary for merging this and thanks @szehon-ho @RussellSpitzer for the review.
Yeah, there is no way to backport it. |
|
@aokolnychyi do you have any suggestion ? |
|
I can create a branch from the tag. |
|
@szehon-ho, could you raise a discussion thread on the dev list for 1.3.1? |
…xception (apache#7931) (cherry picked from commit d89ba12)
* Hive: Set commit state as Unknown before throwing CommitStateUnknownException (apache#7931) (apache#8029) * Spark 3.4: WAP branch not propagated when using DELETE without WHERE (apache#7900) (apache#8028) * Core: Include all reachable snapshots with v1 format and REF snapshot mode (apache#7621) (apache#8027) * Spark 3.3: Backport 'WAP branch not propagated when using DELETE without WHERE' (apache#8033) (apache#8036) * Flink: remove the creation of default database in FlinkCatalog open method (apache#7795) (apache#8039) * Core: Handle optional fields (apache#8050) (apache#8064) * Core: Handle allow optional fields We expect: - current-snapshot-id - properties - snapshots to be there, but they are actually optional. * Use AssertJ * Core: Abort file groups should be under same lock as committerService (apache#7933) (apache#8060) * Spark 3.4: Fix rewrite_position_deletes for certain partition types (apache#8059) * Spark 3.3: Fix rewrite_position_deletes for certain partition types (apache#8059) (apache#8069) * Spark: Add actions for disater recovery. * Fix the compile error. * Fix merge conflicts and formatting * All tests are working and code integrated with Spark 3.3 * Fix union error and snapshots test * Fix Spark broadcast error * Add RewritePositionDeleteFilesSparkAction --------- Co-authored-by: Eduard Tudenhoefner <etudenhoefner@gmail.com> Co-authored-by: Fokko Driesprong <fokko@apache.org> Co-authored-by: Xianyang Liu <liu-xianyang@hotmail.com> Co-authored-by: Szehon Ho <szehon.apache@gmail.com> Co-authored-by: Yufei Gu <yufei_gu@apple.com> Co-authored-by: yufeigu <yufei@apache.org> Co-authored-by: Laith Alzyoud <laith.alzyoud@revolut.com> Co-authored-by: vaultah <4944562+vaultah@users.noreply.github.com>
We should set the
commitStatustoUNKNOWNbefore throwing theCommitStateUnknownExceptionforLockException. Otherwise thecommitStatusis stillFAILUREwhich could lead to the metadata being deleted incleanupMetadataAndUnlock.