[bug]: LND crashes on channel.db compact attempt - after file system crash #8033

dunkfan · 2023-09-19T17:58:04Z

dunkfan
Sep 19, 2023

After a hardware crash due to disk failure I cannot get lnd running again
There is a chance this is not related to the crash because the channel.db file was last written before the crash time
I would appreciate some advice before starting again from scratch

Should I perhaps try to start without compacting?
Or use the chantools program?
Or skip to just doing a scratch and using the SCB file

Environment

0.16.4
Linux raspberrypi 6.1.21-v8+ aarch64 GNU/Linux
btc core 24.0.1

Steps to reproduce

Starting up with automatic compact - the previous working lnd.conf setup - gives this stack trace:

2023-09-19 19:32:19.916 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=true
2023-09-19 19:32:19.917 [INF] CHDB: Compacting database file at /home/ln2/.lnd/data/graph/mainnet/channel.db
unexpected fault address 0xecaf0008
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0xecaf0008 pc=0x63c6e0]

goroutine 1 [running]:
runtime.throw({0x19222f7, 0x5})
runtime/panic.go:1047 +0x4c fp=0x405d18c sp=0x405d178 pc=0x4df3c
runtime.sigpanic()
runtime/signal_unix.go:834 +0xa8 fp=0x405d1a8 sp=0x405d18c pc=0x67fb0
go.etcd.io/bbolt.(*elemRef).isLeaf(...)
go.etcd.io/bbolt@v1.3.6/cursor.go:387
go.etcd.io/bbolt.(*Cursor).first(0x405d268)
go.etcd.io/bbolt@v1.3.6/cursor.go:170 +0x58 fp=0x405d1dc sp=0x405d1ac pc=0x63c6e0
go.etcd.io/bbolt.(*Cursor).next(0x405d268)
go.etcd.io/bbolt@v1.3.6/cursor.go:234 +0x54 fp=0x405d200 sp=0x405d1dc pc=0x63cae4
go.etcd.io/bbolt.(*Cursor).Next(0x405d268)
go.etcd.io/bbolt@v1.3.6/cursor.go:75 +0x30 fp=0x405d244 sp=0x405d200 pc=0x63c008
go.etcd.io/bbolt.(*Bucket).ForEach(0x41f8540, 0x405d2b0)
go.etcd.io/bbolt@v1.3.6/bucket.go:389 +0xd0 fp=0x405d278 sp=0x405d244 pc=0x63a080
github.com/lightningnetwork/lnd/kvdb.(*compacter).walkBucket(0x405d6a0, 0x41f8540, {0x0, 0x0, 0x0}, {0xe22ff3ec, 0xa, 0xa}, {0x0, 0x0, ...}, ...)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/bolt_compact.go:273 +0x178 fp=0x405d2cc sp=0x405d278 pc=0x101cbe8
github.com/lightningnetwork/lnd/kvdb.(*compacter).walk.func1.1({0xe22ff3ec, 0xa, 0xa}, 0x41f8540)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/bolt_compact.go:226 +0x150 fp=0x405d324 sp=0x405d2cc pc=0x101c888
go.etcd.io/bbolt.(*Tx).ForEach.func1({0xe22ff3ec, 0xa, 0xa}, {0x0, 0x0, 0x0})
go.etcd.io/bbolt@v1.3.6/tx.go:129 +0x70 fp=0x405d344 sp=0x405d324 pc=0x64a1c0
go.etcd.io/bbolt.(*Bucket).ForEach(0x3c1e68c, 0x405d38c)
go.etcd.io/bbolt@v1.3.6/bucket.go:390 +0x114 fp=0x405d378 sp=0x405d344 pc=0x63a0c4
go.etcd.io/bbolt.(*Tx).ForEach(0x3c1e680, 0x405d3ac)
go.etcd.io/bbolt@v1.3.6/tx.go:128 +0x58 fp=0x405d398 sp=0x405d378 pc=0x64a128
github.com/lightningnetwork/lnd/kvdb.(*compacter).walk.func1(0x3c1e680)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/bolt_compact.go:220 +0x54 fp=0x405d3b8 sp=0x405d398 pc=0x101c710
go.etcd.io/bbolt.(*DB).View(0x3d6c3c0, 0x405d400)
go.etcd.io/bbolt@v1.3.6/db.go:772 +0xa4 fp=0x405d3ec sp=0x405d3b8 pc=0x640498
github.com/lightningnetwork/lnd/kvdb.(*compacter).walk(0x405d6a0, 0x3d6c3c0, 0x405d444)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/bolt_compact.go:219 +0x54 fp=0x405d40c sp=0x405d3ec pc=0x101c694
github.com/lightningnetwork/lnd/kvdb.(*compacter).compact(0x405d6a0, 0x3d6c640, 0x3d6c3c0)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/bolt_compact.go:147 +0xf8 fp=0x405d460 sp=0x405d40c pc=0x101c170
github.com/lightningnetwork/lnd/kvdb.(*compacter).execute(0x405d6a0)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/bolt_compact.go:117 +0x5b8 fp=0x405d5ac sp=0x405d460 pc=0x101bb48
github.com/lightningnetwork/lnd/kvdb.compactAndSwap(0x405d918)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/backend.go:179 +0x760 fp=0x405d6c4 sp=0x405d5ac pc=0x101ab10
github.com/lightningnetwork/lnd/kvdb.GetBoltBackend(0x405d918)
github.com/lightningnetwork/lnd/kvdb@v1.4.1/backend.go:101 +0x98 fp=0x405d70c sp=0x405d6c4 pc=0x101a090
github.com/lightningnetwork/lnd/lncfg.(*DB).GetBackends(0x40422d0, {0x1ccfd1c, 0x4043440}, {0x3c4ca80, 0x21}, {0x410c150, 0x29}, {0x3c4cab0, 0x2e}, 0x0, ...)