Proper setup for multiple redundant border routers

[Vijay-Karthick]

Hi all,

I have a thread network with two (or more) border routers (nRF52840 dongles running ot-rcp and connected to Raspberry Pi running latest fork of otbr-agent). When I initially setup the network and add an MTD (nRF52840 dev kit running ot-cli-mtd firmware), I'm able to ping global IPv6 addresses (like Google DNS or other laptops in my LAN network). For this, I had to change the IPv6 prefix of Tayga to 2001:db8:1:ffff::/96.

Device 1 = leader
Device 2 = router
Device 3 = MTD

Here, Device 3 is the direct child of Device 1 (based on info from child table).

See image below for more details on the setup:
https://freeimage.host/i/SyQNPn

Ping from Device 3 to ping 2001:db8:1:ffff::8.8.8.8
Logs from Device 1: (tcpdump -i nat64)
17:10:47.053814 IP6 fd11:22::xxxx > 2001:db8:1:ffff::808:808: ICMP6, echo request, seq 1, length 16
17:10:47.053875 IP 192.168.255.42 > dns.google: ICMP echo request, id 1, seq 1, length 16
17:10:47.060735 IP dns.google > 192.168.255.42: ICMP echo reply, id 1, seq 1, length 16
17:10:47.060758 IP6 2001:db8:1:ffff::808:808 > fd11:22::xxxx: ICMP6, echo reply, seq 1, length 16

Logs from Device 2: (tcpdump -i nat64)
No logs

With this setup, if I bring down Device 2 (by stopping otbr-agent), I'm no longer able to ping IPv6 addresses.
https://freeimage.host/i/SyQLKu

Also, I see something really strange. Tcpdump logs on Device 2 shows that the ping from the Device 3 is trying to route via this Device 3.

Ping from Device 3 to ping 2001:db8:1:ffff::8.8.8.8

Logs from Device 1: (tcpdump -i nat64)
17:36:51.919117 IP6 fd11:22::xxxx > 2001:db8:1:ffff::808:808: ICMP6, echo request, seq 7, length 16
17:36:51.919184 IP 192.168.255.42 > dns.google: ICMP echo request, id 7, seq 7, length 16
17:36:51.925831 IP dns.google > 192.168.255.42: ICMP echo reply, id 7, seq 7, length 16
17:36:51.925902 IP6 2001:db8:1:ffff::808:808 > fd11:22::xxxx: ICMP6, echo reply, seq 7, length 16

Logs from Device 2: (tcpdump -i nat64)
17:36:51.924599 IP6 fdbe:e79d:d6e4:0:yyyy > 2001:db8:1:ffff::808:808: ICMP6, destination unreachable, unreachable route fd11:22::xxxx, length 64

*Highlighted addresses (fd11:22::xxxx) in the last few logs are the same.

How do I setup multiple border routers to redundantly work on a single thread network?

CC: @jwhui

Kind regards,
Vijay

[jwhui]

If the MTD is attached to Device 3, disabling Device 2 should not affect the behavior.

On each Thread device, can you provide the output of:

• ipaddr
• neighbor table
• netdata show

These commands should the Thread network topology and the external route configuration.

[Vijay-Karthick]

Is it not advisable to use same ipv6 prefix in all border routers?

[jwhui]

It depends on what prefix you are referring to. Because NAT64 is stateful, the NAT64 prefix should be unique to each border router providing the NAT64 function. However, every Thread device in the Thread network can utilize the same on-mesh prefix for assigning addresses.

[Vijay-Karthick]

I was using the same NAT64 prefix 2001:db8:1:ffff::\96 in all my border routers in /etc/tayga.conf

I observed this issue when I have the same on-mesh prefix (prefix add fd11:22::\64 pasor) in all my border routers. However, I noticed that when I add different on-mesh prefixes (fd11:22::, fd11:33::, fd11:44::, etc.), I did not see this issue.

[jwhui]

Because Thread External Routes are based on source-destination routing, having a different on-mesh prefix for each BR will result in the Thread device selecting a single egress BR based on the IPv6 source address that it chose. Having a different NAT64 prefix for each BR is preferred since the destination selects which stateful NAT64 function a given packet should be routed to.

[Vijay-Karthick]

Thank you for the suggestion, Jonathan. I'll implement this fix and revert back today.