-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
udb: add upgradeToVersion12 #1126
base: master
Are you sure you want to change the base?
udb: add upgradeToVersion12 #1126
Conversation
Few issues: A long time ago we had several different components (address manager, tx store, stake store) all being saved in the database, but they were versioned separately and one component could not use data from another because it could not assume the data format. This was fixed by moving everything under the udb package and merging all of the components under a single database version. The old upgrade code for each individual component was still necessary to bring it up to the state where the merge could be performed, but following that, no more upgrades are made to the components individually. See upgrades.go, upgrades_test.go, and the testdata directory for the new upgrade mechanism and testing infrastructure. Second, you can not delete or put values when using a cursor or within a ForEach function (which also wraps a cursor). This invalidates the cursor and further cursor usage can silently cause database corruption. Read all values which need changes first and then modify them at the end. |
Also, be aware of golang/go#18007 when generating the testdata files. I have needed to rename testdata to something else just to |
I didn't get why to generate new testdata since it is only data inconsistency fix. It should be fully compatible with v7 version, isn't it? |
wallet/udb/upgrades.go
Outdated
// removeMultisigOutsWithoutTransactions is the tenth version of the database. | ||
// It fixes issue with MultisigOuts without related transactions. Simply find all | ||
// detached MultisigOutputs and remove | ||
removeMultisigOutsWithoutTransactionsVersion = 10 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This name is a bit long. removeExtraMultisigOutsVersion
reads better to me. The comment should also describe why they were an issue (failure to insert transactions due to multisig output double spend checks, because they were not always correctly removed in previous database code). And please don't use "simply" when describing the fix, just describe how the fix is performed.
wallet/udb/upgrades.go
Outdated
metadataBucket := tx.ReadWriteBucket(unifiedDBMetadata{}.rootBucketKey()) | ||
txmgrBucket := tx.ReadWriteBucket(wtxmgrBucketKey) | ||
|
||
// Assert this function is only called on version 8 databases. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
9
wallet/udb/upgrades.go
Outdated
|
||
var multisigOutsMarkedForDelete [][]byte | ||
multisigOutBucket := txmgrBucket.NestedReadWriteBucket([]byte("ms")) | ||
multisigOutBucket.ForEach(func(k, v []byte) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer a cursor to be used here for consistency, instead of a ForEach. See the other upgrade functions, which all use cursors.
wallet/udb/upgrades_test.go
Outdated
@@ -15,6 +15,7 @@ import ( | |||
"path/filepath" | |||
"testing" | |||
|
|||
"errors" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
move this to the above group with all of the stdlib packages
wallet/udb/upgrades_test.go
Outdated
@@ -34,6 +35,7 @@ var dbUpgradeTests = [...]struct { | |||
// No upgrade test for V7, it is a backwards-compatible upgrade | |||
{verifyV8Upgrade, "v7.db.gz"}, | |||
// No upgrade test for V9, it is a fix for V8 and the previous test still applies | |||
{verifyV10Upgrade, "v7.db.gz"}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we are testing a 9 -> 10 upgrade, we should have a version 9 database with the error.
After upgrading, the database will be compatible with the older db test artifacts, however those won't expose the problem we are trying to fix here. |
Hi @jrick . I was exploring methods to create mock objects in multisigOut bucket and didn't find any public functions for it. Could you clarify that thing and maybe suggest some approach you would like to see? |
@jrick Travis CI error log doesn't seem like related to new code |
The error is fixed on master now. As for creating the test db, you can just write keys/values directly to the DB. This is fine for this case since we are intentionally trying to create an invalid DB that will be fixed during the upgrade. |
Is it ok If I will leave more complex solution for invalid DB creation? I didn't manage to create valid value of multisig record without using multisig methods |
The invalid DB for the test can be created in a more simpler way by using keys and values created from the |
Changed approach to use |
eba6b21
to
3e900aa
Compare
wallet/udb/upgrades.go
Outdated
Height: int32(mso.BlockHeight), | ||
}) | ||
if txVal == nil { | ||
multisigOutsMarkedForDelete = append(multisigOutsMarkedForDelete, k) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to be on the safe side, make a copy of the key. bolt is unfortunately ridden with bugs related to cursor invalidation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jrick could you check this change in fresh version. Did I make it right?
e57e14c
to
4072ab6
Compare
6bde0c1
to
17d79ba
Compare
In this upgrade function we do check all MSOs for related tx existence. If TX doesn't exist we drop MSO
17d79ba
to
0ce1ec7
Compare
Done |
#1036
In this upgrade function we do check all MSOs for related
tx existence. If TX doesn't exist then we drop MSO