LND stopped working after restart (stuck in a loop)

[guitarchair]

I just restarted my node and lnd stopped working, the logs show me this:

2024-07-24 10:49:14.274 [INF] LTND: Version: 0.18.0-beta commit=v0.18.0-beta, build=production, logging=default, debuglevel=info
2024-07-24 10:49:14.274 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2024-07-24 10:49:14.275 [INF] RPCS: RPC server listening on 0.0.0.0:10009
2024-07-24 10:49:14.283 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080
2024-07-24 10:49:14.283 [INF] LTND: Opening the main database, this might take a few minutes...
2024-07-24 10:49:14.283 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false
2024-07-24 10:50:06.026 [WRN] LTND: Config 'bitcoin.active' is deprecated, please remove it
2024-07-24 10:50:06.026 [INF] LTND: Version: 0.18.0-beta commit=v0.18.0-beta, build=production, logging=default, debuglevel=info
2024-07-24 10:50:06.026 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2024-07-24 10:50:06.027 [INF] RPCS: RPC server listening on 0.0.0.0:10009
2024-07-24 10:50:06.032 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080
2024-07-24 10:50:06.032 [INF] LTND: Opening the main database, this might take a few minutes...
2024-07-24 10:50:06.032 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false

[guitarchair]

I now tried to copy the backup of the /data dir I have (basically deleted the node config I had), and now I can at least start the lnd app. Maybe it will solve the issue after I try to restart again (I'll try that, after it's done rescanning).
I'll update this post later on with the results.

[guitarchair]

I only have 4 apps installed: Bitcoin, electrs, lnd, RTL (ride the lightning).
I tried stopping RTL and restart LND but I still get the same stuck loop.
Bitcoind and electrs work fine and 100% sync.
What else could I check that's shutting down lnd? I mean what tool/command could be useful to troubleshoot the source of the problem?

[guggero]

Hmm, I don't really have a good idea... Since it kind of stops after trying to open the main database, you could do a manual check if it is healthy. You can do that by stopping lnd, then running chantools compactdb on it (either on Umbrel itself or by copying the /data/umbrel-os/home/umbrel/umbrel/app-data/lightning/data/lnd/data/graph/mainnet/channel.db file to another machine and running chantools there.

[guitarchair]

Since the same problem is happening with both the older database and new fresh database, and in both cases the database works fine right after the installation, but not after a restart, it doesn't seem like a corrupt database file problem...
I'm thinking to perhaps uninstall LND from the web GUI and then install it from cli (the regular lnd version, not the umbrel version). Is that something that might work? I'm just unsure if will cause other problems, like using RTL for opening channels (because RTL is using a web GUI).
I'm also adding the docker inspect file in case that can shed more light. Notice the cmd part in it, maybe these options can give a clue:
"Cmd": [
"--configfile=/data/.lnd/umbrel-lnd.conf",
"--listen=0.0.0.0:9735",
"--rpclisten=0.0.0.0:10009",
"--restlisten=0.0.0.0:8080",
"--bitcoin.active",
"--bitcoin.mainnet",
"--bitcoin.node=bitcoind",
"--bitcoind.rpchost=10.21.21.8:8332",
"--bitcoind.rpcuser=umbrel",
"--bitcoind.rpcpass=REDUCTED PASSWORD",
"--bitcoind.zmqpubrawblock=tcp://10.21.21.8:28332",
"--bitcoind.zmqpubrawtx=tcp://10.21.21.8:28333",
"--tor.active",
"--tor.v3",
"--tor.control=10.21.21.11:29051",
"--tor.socks=10.21.21.11:9050",
"--tor.targetipaddress=10.21.21.9",
"--tor.password=REDUCTED PASSWORD="
],

docker_inspect_lnd.txt

[guitarchair]

I found some more interesting logs related to the bbolt database, maybe you can make sense of it?
In the inspect log I found the "log path" which has more info in it:
"LogPath": "/var/lib/docker/containers/8ced8f733de32a96af9b067e814ca13e3053395dc8a8565d69a75bb737a6b07f/8ced8f733de32a96af9b067e814ca13e3053395dc8a8565d69a75bb737a6b07f-json.log"

This is part of the output. Notice the "page" error related to bbolt. So maybe it is a file corruption after all?
I'm going to simply uninstall LND (again), and reinstall and hopefully this time there won't be a db corruption. I'd rather do that then start dealing with compaction of a file that really has nothing on it except for the seed. There are no channels opened yet.

{"log":"2024-07-25 17:10:26.259 [INF] LTND: Active chain: Bitcoin (network=mainnet)\n","stream":"stdout","time":"2024-07-25T17:10:26.259349395Z"}
{"log":"2024-07-25 17:10:26.260 [INF] RPCS: RPC server listening on 0.0.0.0:10009\n","stream":"stdout","time":"2024-07-25T17:10:26.26071092Z"}
{"log":"2024-07-25 17:10:26.275 [INF] RPCS: gRPC proxy started at 0.0.0.0:8080\n","stream":"stdout","time":"2024-07-25T17:10:26.275169707Z"}
{"log":"2024-07-25 17:10:26.275 [INF] LTND: Opening the main database, this might take a few minutes...\n","stream":"stdout","time":"2024-07-25T17:10:26.27546649Z"}
{"log":"2024-07-25 17:10:26.275 [INF] LTND: Opening bbolt database, sync_freelist=false, auto_compact=false\n","stream":"stdout","time":"2024-07-25T17:10:26.275739418Z"}
{"log":"2024-07-25 17:10:26.286 [INF] LTND: Shutdown complete\n","stream":"stdout","time":"2024-07-25T17:10:26.286297227Z"}
{"log":"\n","stream":"stdout","time":"2024-07-25T17:10:26.286313514Z"}
{"log":"panic: assertion failed: Page expected to be: 3436, but self identifies as 9552306040493850340\n","stream":"stderr","time":"2024-07-25T17:10:26.288428026Z"}
{"log":"\n","stream":"stderr","time":"2024-07-25T17:10:26.2884656Z"}
{"log":"goroutine 1 [running]:\n","stream":"stderr","time":"2024-07-25T17:10:26.288474502Z"}
{"log":"go.etcd.io/bbolt._assert(...)\n","stream":"stderr","time":"2024-07-25T17:10:26.288486591Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/db.go:1359\n","stream":"stderr","time":"2024-07-25T17:10:26.288493342Z"}
{"log":"go.etcd.io/bbolt.(*page).fastCheck(0x7f2825b6c000, 0xd6c)\n","stream":"stderr","time":"2024-07-25T17:10:26.288499766Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/page.go:57 +0x1d9\n","stream":"stderr","time":"2024-07-25T17:10:26.288508667Z"}
{"log":"go.etcd.io/bbolt.(*Tx).page(0x334cd28?, 0x1b79dc0?)\n","stream":"stderr","time":"2024-07-25T17:10:26.288515937Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx.go:534 +0x7b\n","stream":"stderr","time":"2024-07-25T17:10:26.288522199Z"}
{"log":"go.etcd.io/bbolt.(*Tx).forEachPageInternal(0xc00017a9a0, {0xc00096c5a0, 0x4, 0xa}, 0xc00079c588)\n","stream":"stderr","time":"2024-07-25T17:10:26.288528569Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx.go:546 +0x5d\n","stream":"stderr","time":"2024-07-25T17:10:26.288535452Z"}
{"log":"go.etcd.io/bbolt.(*Tx).forEachPageInternal(0xc00017a9a0, {0xc00096c5a0, 0x3, 0xa}, 0xc00079c588)\n","stream":"stderr","time":"2024-07-25T17:10:26.288543722Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx.go:555 +0xc8\n","stream":"stderr","time":"2024-07-25T17:10:26.288550472Z"}
{"log":"go.etcd.io/bbolt.(*Tx).forEachPageInternal(0xc00017a9a0, {0xc00096c5a0, 0x2, 0xa}, 0xc00079c588)\n","stream":"stderr","time":"2024-07-25T17:10:26.288557221Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx.go:555 +0xc8\n","stream":"stderr","time":"2024-07-25T17:10:26.288564665Z"}
{"log":"go.etcd.io/bbolt.(*Tx).forEachPageInternal(0xc00017a9a0, {0xc00096c5a0, 0x1, 0xa}, 0xc00079c588)\n","stream":"stderr","time":"2024-07-25T17:10:26.28863353Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx.go:555 +0xc8\n","stream":"stderr","time":"2024-07-25T17:10:26.288645391Z"}
{"log":"go.etcd.io/bbolt.(*Tx).forEachPage(...)\n","stream":"stderr","time":"2024-07-25T17:10:26.288652871Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx.go:542\n","stream":"stderr","time":"2024-07-25T17:10:26.28865892Z"}
{"log":"go.etcd.io/bbolt.(*Tx).checkBucket(0xc00017a9a0, 0xc000934bc0, 0xc00079c990, 0xc00079c8d0, {0x238aea0, 0x350cd40}, 0xc0009984e0)\n","stream":"stderr","time":"2024-07-25T17:10:26.288694638Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx_check.go:83 +0x114\n","stream":"stderr","time":"2024-07-25T17:10:26.288703585Z"}
{"log":"go.etcd.io/bbolt.(*Tx).checkBucket.func2({0x7f2826ffd3bb?, 0xc00079c6c0?, 0xc00044f840?})\n","stream":"stderr","time":"2024-07-25T17:10:26.288710251Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/tx_check.go:110 +0x90\n","stream":"stderr","time":"2024-07-25T17:10:26.2887169Z"}
{"log":"go.etcd.io/bbolt.(*Bucket).ForEachBucket(0x0?, 0xc00079c788)\n","stream":"stderr","time":"2024-07-25T17:10:26.288724261Z"}
{"log":"\u0009go.etcd.io/bbolt@v1.3.7/bucket.go:403 +0x96\n","stream":"stderr","time":"2024-07-25T17:10:26.28873068Z"}
{"log":"go.etcd.io/bbolt.(*Tx).checkBucket(0xc00017a9a0, 0xc00017a9b8, 0xc00079c990, 0xc00079c8d0, {0x238aea0, 0x350cd40}, 0xc0009984e0)\n","stream":"stderr","time":"2024-07-25T17:10:26.288749326Z"}

The full log file here:
container_log_lnd.txt

[guggero]

So maybe it is a file corruption after all?

Yeah, that's what those errors mean unfortunately. So either it was interrupted unexpectedly (power loss?) or something's not good with the power supply to the hard drive (this sometimes happens with Raspberry Pis, bad power supply issues).

[guitarchair]

Okay I was able to restart the node and now LND is able to start... hopefully this time it won't happen again... or maybe it's better to save a backup of the database and then simply replace a corrupt file.

[guggero]

better to save a backup of the database and then simply replace a corrupt file.

Not if you want to put your channels at risk! Didn't you read the link I sent you above? Channels force closing is just the "optimal" case for running with an old channel state. The worst case is you also lose all funds in those channels. So no, definitely never restore a channel.db file from a backup!

[guitarchair]

Right, I meant the channel.backup file specifically. Although it's unclear to me, if the file itself is already the backup, why do I need to back it up? And if it is a backup, why couldn't I just use that file instead of uninstalling LND completely and reinstalling it? Or is the "bbolt" database referring to other files?

[guggero]

The role of the channel.backup file (also known as the SCB file, "Static Channel Backup") is also described in that document.
The idea of backing that file up is to have it in a separate location in case the whole disk is defective.
BUT, the SCB only contains a channel shell, enough information to connect out to your peers and ask them to force close. So you're fully relying on them to close the channel. If they aren't around anymore, you can't get your funds back.
That's why it's so important to make sure the channel.db (which is the bbolt database) is healthy and doesn't corrupt.

So if you plan on having a non-trivial amount on your node, I'd suggest to have a bit more of a sturdy disk setup than an external, USB connected drive, as that's quite unreliable (just speaking from experience of other node runners having similar difficulties).

[guitarchair]

Thanks for the explanation and the help.
If the node fails again I might have to change hardware to internal SSD.