Core Lightning is incompatible with the Lightning Network - lacks support for payments #7180
Comments
|
To clarify, it is uncertain if this is a problem wiht my node or something to do with core lightning. |
|
What happens if you put |
|
@vincenzopalazzo I was away from computer but I can try that now. Before I do that I want to know what that setting does. I am not finding any documentation on it. What is that setting? |
|
the setting disable the multi path payment algorithm |
|
disable-mmp is not a recgonized command or config setting by my Core Lightning. |
|
@nakoshi-satamoto is |
Did this change anything for you? @nakoshi-satamoto |
|
@kilrau I actually did not need to. Ever since I upgraded to core lightning verison 24.02 I kept getting alot of these errors in my logs for various channels. This also was an issue on V24.02.1
After updating to the latest version of core lightning onto version 43.02.2 those gossip issues went away, transactions inbound and outbound are not failing anymore. Maybe it was related to gossip not working? |
probably yes |
|
No longer an issue |
|
Did 24.02.2 fix this for you? I upgraded to that and I'm still having the same issue. (Pages of |
|
On 2024-05-05 18:52, BitcoinMechanic wrote:
Did 24.02.2 fix this for you? I upgraded to that and I'm still having
the same issue. (Pages of `Bad gossip order....`)
I am on 24.02.2 and I still get those bad gossip order messages. I am
able to do payments though.
|
|
also on 24.02.02 with high - payment failure rate and bad gossip order messages |
|
I'm also on 24.02.02 (using the BTCPay Docker image), and 100% payment failures unless paying direct channel partners, and tons of bad gossip order messages. Cannot query remote routes at all, even to targets that are direct partners of channel partners. Incoming payments and some through-routes seem to work though (the through routes may only be working to next-node termination though). Example log entries:
UPDATE: |
|
I said that I can still do payments, I am ammending my claim. I can receive inbound payments no problem but I do have a high rate of failure for outbound payments unless it is a direct peer. Ever since I upgraded to version 24 I have noticed this and it also coinciding with the gossip error messages. I delete gossip database and it still keeps getting gossip errors. |
|
I am also experiencing this exact issue. |
|
DO NOT delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill. Are you missing gossip? A healthy node should look like this: That is, about 85000 public channels and 9000 nodes. |
|
Well… in an attempt to make my core lightning node functional again I did delete the gossip store (but did save a backup! (This was… a week ago?)) should I just continue as is or cp gossip_store.backup to replace the “new” one? Have you not noticed any payment issues Rusty?
On Mon, Jun 3, 2024, at 9:26 PM, Rusty Russell wrote:
*DO NOT* delete the gossip store, you will not be able to make payments any more. You've made the network disappear, and it can take a while to refill.
Are you missing gossip? A healthy node should look like this:
`$ lightning-cli -H listchannels | grep -c ^short_channel_id
85921
$ lightning-cli -H listnodes | grep -c ^nodeid
9062
`
… That is, about 85000 public channels and 9000 nodes.
—
Reply to this email directly, view it on GitHub <#7180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AND3BCSLAHAFJBKAP3S2W4LZFU6XHAVCNFSM6AAAAABFLB774SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBWGU3TSNZXGM>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
On 2024-06-04 06:26, Rusty Russell wrote:
*DO NOT* delete the gossip store, you will not be able to make
payments any more. You've made the network disappear, and it can take
a while to refill.
Are you missing gossip? A healthy node should look like this:
```
$ lightning-cli -H listchannels | grep -c ^short_channel_id
85921
$ lightning-cli -H listnodes | grep -c ^nodeid
9062
```
That is, about 85000 public channels and 9000 nodes.
What is wrong with deleting gossip? I have done that often times in
version 24 to fix gossip issues. It re-populates fast.
I deleted gossip yesterday after I upgraded to version 24.05 which
claims to fix the gossip issues.
I just checked and I have 92910 channels, 16063 nodes.
|
|
Did this resolve your issue with payments ? |
|
On 2024-06-07 02:33, blapcity wrote:
Did this resolve your issue with payments ?
I had the opportunity to test some payments today. I have 12 chans and
almost 10,000,000 of outbound liquidity combined. Many of the peers I'm
connected to are well connected also. I tried three random payments of
less than 100,000 to different destinations and all of them failed. I
still see gossip errors sometimes also, even after upgrading to 24.05
and rebuliding gossip. I'd say my gossip database is healthy also as I
have 14427 nodes, 92599 chans in my view.
As of now, Core Lightning _cannot_ send payments unless it is a directly
connected peer. This is not functional.
|
|
On 2024-06-10 19:46, Satoshi wrote:
I had the opportunity to test some payments today. I have 12 chans and
almost 10,000,000 of outbound liquidity combined.
10 millions sats.
|
|
Any chance we could get some more logs to try and hone in on the cause for the increased failure rate? From the above they all seem to come more or less directly after deleting gossip_store (fails immediately due to lack of network view, rather than taking some time and performing real attempts). The gossip order is very likely to be unrelated, so ignore that maybe for a bit. It is mostly a warning that the peer has given us gossip messages in the incorrect order (update before announcement, or node before any of it's channels). |
|
One more thing, how many here are running btcpayserver? As it has been mentioned a number of times, I'd like to exclude that as a potential source. |
|
I have similar problems since upgrading to 24.05 without running btcpayserver. I'm only connected to boltz with a channel capacity of 1.000.000 sats. The logs below are from a failed transaction to phoenix wallet. Sending funds to wallet of satoshi was also not possible (same amount).
If this issue is related, I'm happy to share more information. |
|
Ok, I think I see what's happening. Just to be sure, can you provide the couple more lines above the snippet you posted? It contains the channel hints that we initialize based on our local view of the channels ( Thanks for the logs, they are incredibly helpful 👍 |
|
Also, please set your log levels to debug. |
|
Thank your for taking the time. I set log levels to debug and did three transactions. 1 sat --> succeeded cln_debug_failed_1000sats_primal.txt |
|
On 2024-08-07 14:07, Lars Hagström wrote:
> I can do some tests. How do I see what route a payment is trying to
> take?
I saw peer ids in my system logs, and then I excluded them with the
command line option, like:
`lightning-cli pay bolt11=<invoice> exclude=["<66 char id>","<66 char
id>"]`
Thank you. So the issue does seem to be that when payment fails, it is
due to a certain channel that Core Lightning is attempting, and the
issue is that it does not re-attempt over other channels. So a work
around indeed may be to when a payment fails, refer to the log to
identify which channel it was failing to send over, then re-attempt
payment again with excluding that channel that is failing the payment.
And of course to alternatively also upload debug logs to help devs
investigate the issue.
So we now know a work around,
A simple fix may be to just have the pay command attempt payment over
more than just one channel in the meantime until the root cause of the
issue is addressed.
|
|
A solution is already be worked on: |
|
I wanted to confirm a work-around that others have mentioned. If a payment fails I look at the log to see what channel the pay plugin was attempting to send it through, I then exclude that channel on my next payment attempt and it goes through fine. |
|
Now even when I exclude channels payments are failing, regardless of payment size. I cannot even simply do a little 8,000 sat payment. Excluding one or more channels does nothing to resolve this. And this is not fixed in the next release, because the next release, v24.08rc2 does not even start up. #7595 As of now, Version 24 does not work and there is no work around. The only solution at this point is to close all channels in Core Lightning and switch to something else, or wiat for a future release that might fix this issue. v24.08rc2 does not work at the moment. |
|
I think I am closing down my Core Lightning node sadly, now I have to re-program things to work with a different lightning implementation. This goes back to my question, what other options are there? Is LND the only option? I'd rather avoid go-lang if possible. And please do not advise anything written in python. |
nakoshi-satamoto
changed the title
|
I renamed this issue to "Core Lightning is incompatible with the Lightning Network - lacks support for payments" because a fundamental trait of the lightning network is the ability to do payments. Version 24 discontinued support for payments and so I consider it no longer a lightning network compatible implementation. At this point I'm considering Core Lightning no longer a legitimate option. Seeking alternatives. I apprecaite devs and the work that has been behind Core Lightning for the many years, maybe one day this issue will be fixed. In the meantime I am seeking alternatives or I will fork Core Lightning v23 into a new project to bring back compatibility with the lightning network again. Core Lightning v23 is the last functioning version. Maybe at this point it would be good to just delete all commits in version 24 and go back to v23? |
@kilrau How can you say that "nothing comes close" when CLN does not even support payments? Isn't the ability to do payments the main purpose of the lightning network? I really am curious how you are even getting CLN to work if you are using CLN. I have a well established node but I am closing all my channels tonight to switch to something else (LND is maybe the only option at this point, as much as I hate go-lang) or I am going to fork CLN from versoin 23. |
Well we were as frustrated as you were so @michael1011 wrote |
|
I confirm I am getting the same problems with the impossibility to pay any invoice which does not end on my direct peers. I was able to pay an invoice today by using Renepay and excluding one of the two channels. Since the payment was completed, I expect it should be doable automatically from a QR scan, without the need to manually exclude a channel. I am available to provide any details needed to figure out a solution, or to make some tests like paying myself on WoS. Let me know. |
|
I am facing this issue as well when trying to pay to destinations I don't have a direct channel with. renepay works though |
|
I am creating a bounty to fix this problem. Funding at 1AzajWfrD8X6XoJEZ2Vshbd5492B1sxamT. Core Lightning's problem: the pay function fails (#7180). Boltz Exchange wrote a workaround plugin (#6793) but this fix is insufficient. CLN needs a native pay function that works out-of-the box. LND and Phoenix fail to find payment routes at times, but not like CLN, which constantly fails to pay invoices, regardless of the number of nodes connected, channel balance, or invoice amount. Once CLN is fixed and deployed (i.e. once users can use the native pay function reliably without any add-ons/plugins), the funds in the address above will be sent to the developers who are responsible for the fix. If multiple developers contribute to the solution, the funds will be allocated proportionally, with CLN lead developer Rusty Russell determining the payout proportions. For updates to this bounty, see: https://pastebin.com/raw/iACc2pYv To verify fund ownership: |
enjoy, libplugin-pay.c , Addr: 174139836282ferT1B4ULchkLqvCFujyNM. |
I can't read this code, but if this is the solution, get the CLN developers to incorporate your solution into Core Lightning's codebase to fix CLN's native pay function. Once you have achieved this, please let us know and we'll download the latest CLN package to verify that the pay function has been fixed, and pay you whatever proportion of the allocated funds Rusty Russell specifies. |
|
In case anyone is working on this problem from the Core Lightning team, I will be in Lugano for Plan B Forum. |
|
I have not been using lightning since I hvae closed all my channels a couple months ago. I am curious, is this still an issue even with version 24.08, people who report as still having issues with payments (that are not destined for directly connected peers) are you using the latest most up to date version? I happened to see there was another version in the last 20 hours from the time of thist post, 24.08.2. Thank you everyone and to all the developers. |
|
It is still an issue as of 24.08.1 (latest version currently on StartOS). I'll post an update once 24.08.2 has been pulled in. |
|
I am also waiting on 24.08.2 to test. Currently on 24.08.1 (StartOS) and only have success with small 100 sat payments. I have tried payments of 1000, 5000, and 10,000 SATs and they fail every time. |
|
I just updated my node, so perhaps I can most likely give some more insights in the upcoming days. |
|
I'm experiencing the same issue with CLN on my RaspiBlitz. I'm getting 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE) when trying to pay anyone. I upgraded to v24.08.2 to see if it would resolve it but unfortunately, it didn't. I have an option in the RaspiBlitz to view a CLN node summary and when I select that, it shows num_channels=4, num_connected=4, and num_gossipers=0. It appears that the node isn't gossiping with channel partners at all for some reason? I would assume num_gossipers should be equal to the amount of channels or peers. I also had to delete the gossip store to get CLN to start when it became corrupted from a hard shutdown about a month ago. Perhaps that could have caused this issue? |
|
Let me reply on behalf of the CLN team as they don't seem to be monitoring this issue: they are 100% aware of So the answer to this thread is: |
|
So in v24.08.2 we started remembering information across payments, so you'll see a boost if you send multiple payments in a short timeframe (so that the inferred information isn't stale and can be used). xpay for sure is going to improve this massively. |
|
As we speak, there is a working version of renepay that people could try. For users using a remote control app, or not able to call renepay directly, you could |
|
I just did some testing with renepay from cli and had some success. Thanks for the tip @Lagrang3 |
|
If I've been using |
|
You touched directly on the reason there: renepay is a standalone reimplementation of pay, combining both the planning as well as the execution of a payment. These two, interleaved, steps are rather complex on their own, and reimplementing them the execution side was lacking, hence the crashes. With askrene and xpay, Rusty has taken a step back and noticed that the planning and execution phase do not have to be as tightly coupled. Xpay now implements the execution of the payments, while askrene contains the planning phase. The advantage is that we can tweak and experiment with the planning in askrene, or replace it wholesale, without having to reimplement the execution logic, which was the cause for renepay's lack of stability. TL;DR: renepay is being split apart into xpay and askrene, because it frees us from having to reimplement the execution logic for every experiment. |
and others
I am not sure if it is just me or others also, but I have noticed that ever since updating to version 24 most payments fail.
I get alot of these messages, the htlc and node number varies though.
htlc 16 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)then payment deadline expires.
I have even attempted to pay various invoices even with small payments, I also have some large channels (millions of sats) open with large and well connected nodes.
Posting this for visibility in case if others are experiencing this also. If there is something I can check to confirm if it is an issue with my node or corelightning I can do so.
The text was updated successfully, but these errors were encountered: