Rep crawler switching between IPv4 and IPv6 for certain representatives

[SergiySW]

Summary

If representative node has both IPv4 and IPv6 addresses, then rep crawler will frequently reassign representative to different channels.
As of now, it seems, check for existing node ID is removed when establishing new connections and node can have multiple connections to the same peer.

Additional question I didn't check: can node actually flood same data (blocks propagation etc.) twice to the same peer via both IPv4 and IPv6?

Node version

27.0

Build details

manually built

OS and version

Linux

Steps to reproduce the behavior

1. Launch live network node
2. Search log files "warning" level for "[rep_crawler] [warning] Updated representative:" items

Expected behavior

Don't switch representative channel often

Actual behavior

Frequent changes of representative channel between IPv4 and IPv6

[2024-09-02 17:30:30.170] [rep_crawler] [warning] Updated representative: nano_16u1uufyoig8777y6r8iqjtrw8sg8maqrm36zzcm95jmbd9i9aj5i8abr8u5 at: [2a01:7e00::f03c:91ff:fe7a:c863]:57702 (was at: [::ffff:139.162.199.142]:7075)
...
[2024-09-02 17:30:35.841] [rep_crawler] [warning] Updated representative: nano_16u1uufyoig8777y6r8iqjtrw8sg8maqrm36zzcm95jmbd9i9aj5i8abr8u5 at: [::ffff:139.162.199.142]:7075 (was at: [2a01:7e00::f03c:91ff:fe7a:c863]:57702)

Possible solution

A. This should improve live network node behavior, but probably it's not desired by new network design
if (rep.channel->get_endpoint () != channel->get_endpoint () && rep.channel->get_node_id () != channel->get_node_id ())

B. Make a setting to prefer either IPv4 or IPv6 for this or any similar situations.

C. Group peers by node ID rather than IP and allow multiple connections per one node ID (?). Don't send duplication data to different channels of one peer.

Supporting files

log_2024-09-02_17-23-07_463301168.log

[gr0vity-dev]

Thanks for the report!
Great to have you look into issues again :-)

We previously discussed implementing support for multiple addresses per rep.
This would solve the observed issue and could further improve system resiliency.
The idea wasn't prioritized due to other pressing features.

Given that you've already looked into this quite a bit, would you be interested in helping to implement such functionality?

[SergiySW]

@gr0vity-dev thank you for the response!
I would look into it.

Do you mean a single node with a representative and a few different addresses or rather also availability to use same representative within different nodes? I guess I see it with nano_35btiz1mgfwp95c3ckazmzbp5gepduxtijuijd9xebeau8u1gsbea41smjca switching between 147.135.97.16 and 51.81.243.15

[gr0vity-dev]

You are right, it's not super obvious what the best route might be.

We discussed a bit and thought of self-disabling voting for a rep when it detects a vote signed by it's key from a different node.
However, without authentication, it's not so easy to reliably identify these nodes.
So it might be best to wait for authenticated channels before moving forward with this issue.

Alternatively it's possible to make the rep crawler track multiple channels per rep with some kind of scoring etc. This should be possible even without authenticated channels. However it needs to be well designed.

Maybe there are other more obvious areas that can be improved right now...

[gr0vity-dev]

For completeness,

• A node might not reliably know its own IP address without external services
• node_id can be the same on multiple machines

We discussed a method for a representative to determine if its key is being used by another node if the rep receives a vote signed with its own key but with a timestamp in the future —due to slight clock differences— it indicates that another node is using the same key. Since timestamps are part of the signed confirm_ack (vote) message, the rep could self-disable voting upon detecting this.

However one possible issue with this is NTP adjusting the clock might falsely trigger this detection.