panic with invalid freelist page

[Cardinalinux]

panic: invalid page type: 0: 4
panic: invalid freelist page: 0, page type is meta [recovered]
panic: invalid freelist page: 0, page type is meta
goroutine 228 [running]:
github.com/PPIO/go-ppio/util/sys.Go.func1.1()
/home/workspace/go/src/github.com/PPIO/go-ppio/util/sys/panic.go:55 +0x76
panic(0xd27bc0, 0xc42093a9c0)
/usr/lib/go/src/runtime/panic.go:502 +0x229
go.etcd.io/bbolt.(*freelist).read(0xc4202ef230, 0x7fa4f8655000)
/home/workspace/go/src/go.etcd.io/bbolt/freelist.go:237 +0x2fd
go.etcd.io/bbolt.(*freelist).reload(0xc4202ef230, 0x7fa4f8655000)
/home/workspace/go/src/go.etcd.io/bbolt/freelist.go:297 +0x50
go.etcd.io/bbolt.(*Tx).rollback(0xc4201880e0)
/home/workspace/go/src/go.etcd.io/bbolt/tx.go:267 +0xd0
go.etcd.io/bbolt.(*DB).Update.func1(0xc4201880e0)
/home/workspace/go/src/go.etcd.io/bbolt/db.go:662 +0x3e
panic(0xd27bc0, 0xc42093a9a0)
/usr/lib/go/src/runtime/panic.go:502 +0x229
go.etcd.io/bbolt.(*Cursor).search(0xc4201195f8, 0xc42091dee0, 0xb, 0x10, 0x4)
/home/workspace/go/src/go.etcd.io/bbolt/cursor.go:250 +0x388
go.etcd.io/bbolt.(*Cursor).seek(0xc4201195f8, 0xc42091dee0, 0xb, 0x10, 0x0, 0x0, 0xe27560, 0xc4209122e0, 0xc420119520, 0xc42095afc0, ...)
/home/workspace/go/src/go.etcd.io/bbolt/cursor.go:159 +0xa5
go.etcd.io/bbolt.(*Bucket).CreateBucket(0xc4201880f8, 0xc42091dee0, 0xb, 0x10, 0xc42091deeb, 0x5, 0xcb6400)
/home/workspace/go/src/go.etcd.io/bbolt/bucket.go:165 +0xfa
go.etcd.io/bbolt.(*Bucket).CreateBucketIfNotExists(0xc4201880f8, 0xc42091dee0, 0xb, 0x10, 0xb, 0x10, 0xc42091dee0)
/home/workspace/go/src/go.etcd.io/bbolt/bucket.go:199 +0x4d
go.etcd.io/bbolt.(*Tx).CreateBucketIfNotExists(0xc4201880e0, 0xc42091dee0, 0xb, 0x10, 0xb, 0x10, 0x2)
/home/workspace/go/src/go.etcd.io/bbolt/tx.go:115 +0x4f
github.com/timshannon/bolthold.(*Store).TxInsert(0xc4200c4630, 0xc4201880e0, 0xd27bc0, 0xc42093a990, 0xe27560, 0xc4201baa00, 0x0, 0xc4201198f0)
/home/workspace/go/src/github.com/timshannon/bolthold/put.go:50 +0xc3
github.com/timshannon/bolthold.(*Store).Insert.func1(0xc4201880e0, 0x1009490, 0xc4201880e0)
/home/workspace/go/src/github.com/timshannon/bolthold/put.go:38 +0x58
go.etcd.io/bbolt.(*DB).Update(0xc4203023c0, 0xc420119958, 0x0, 0x0)
/home/workspace/go/src/go.etcd.io/bbolt/db.go:670 +0x90
github.com/timshannon/bolthold.(*Store).Insert(0xc4200c4630, 0xd27bc0, 0xc42093a990, 0xe27560, 0xc4201baa00, 0x0, 0x0)
/home/workspace/go/src/github.com/timshannon/bolthold/put.go:37 +0x8a

[WIZARD-CXY]

the freelist is kinda corrupt maybe I think

[jar349]

How can one recover from this state? My containerd won't start after a power-outage due to:

panic: invalid freelist page: 1113, page type is leaf
goroutine 1 [running]:
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*freelist).read(0xc000018480, 0x7fbacd454000)
       /build/microk8s/parts/containerd/go/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/freelist.go:266 +0x265
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).loadFreelist.func1()

[ptabor]

Please try to run bboltctl check on your DB file:

bbolt/cmd/bbolt/main.go

Line 111 in f6be823

case "check":

[smn-snkl]

Having the same issue. Can't start the docker daemon anymore after a server failure due to

panic: invalid freelist page: 1490838, page type is leaf

goroutine 744 [running]:
github.com/docker/docker/vendor/go.etcd.io/bbolt.(*freelist).read(0xc001868300, 0x7f4a6ef34000)

Where do I need to run bboltctl check? It seems I don't even have bboltctl installed, seems to be handled somehow via Docker.

How can I get my docker environment back up and running without losing any data from my docker containers?!

[lukevalenta]

You can install the bolt command line tool following https://github.com/etcd-io/bbolt/tree/master#installing. The executable is bbolt and should end up in your go path.

[alakesh]

Somewhat related issue: #103

[Xalior]

You can install the bolt command line tool following https://github.com/etcd-io/bbolt/tree/master#installing. The executable is bbolt and should end up in your go path.

Which is great. It checks the bolt meta.db and confirms it's broken:

page 35: reachable freed
page 4: reachable freed
page 54: reachable freed
page 2: unreachable unfreed
...
page 41: unreachable unfreed
38 errors found
invalid value

But that doesn't actually help anyone fix anything. There appears to be no extra switches to check to fix the databases.

Vaguely hoping that a compact would fix things - one tries
bbolt compact -o fixed.db meta.db just to get:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4f0fa9]

leading to no-where...

So - how does one fix a corrupt bbolt database, it's all well and good knowning it's broken, but that's no help if you can't fix it. You may as well just tell people to delete the database and start again...