Network traffic triggers watchdog

[ryan-summers]

While testing out some multicast UDP traffic, I observed booster reboot twice. The service then indicated that a watchdog occurred. It's not clear if this was related to the Multicast traffic or not, but seemed to only occur when executing the Stabilizer streaming HITL with a streaming IP of 224.192.168.1

> service
Version             : Unspecified [release]
Hardware Revision   : v1.5
Rustc Version       : rustc 1.72.0 (5680fa18f 2023-08-23)
Features            :
Detected Phy        : W5500
Panic Info          : None
Watchdog Detected   : true

>

[ryan-summers]

This was running on 1ea1124

[jordens]

n.b.
I'm also streaming multicast in the same net on 239.34.16.10.
I would not use multicast in 224.0.0.0/8. Let's stick to the admin scoped 239.0.0.0/8.

[ryan-summers]

This appears pretty reproducible with 239.192.168.1 in my network. I'm looking into what's happening

[ryan-summers]

Disabling the watchdog indicates that booster remains operational throughout the whole event and doesn't encounter a true lockup. What I believe may be happening is that there's so much traffic incoming on the PHY that smoltcp and the W5500 need to process that it slows down processing of the application in general. I wonder if there's a way we can handle events like this where there's excessive ethernet traffic.

[jordens]

It should not be seeing that traffic at all. The switch doesn't flood.
It might see IGMP traffic but that's not much at all and always has been there.

[ryan-summers]

I see the green ethernet LED remain consistently on during the Stabilizer stream period, indicating that Booster is indeed receiving excessive packets.

I opened smoltcp-rs/smoltcp#848 around this, but this could be due to a cheap switch that isn't handling multicast properly as well.

[jordens]

That doesn't seem to be the case here. I'm seeing close to zero traffic towards stabilizer (while another one is streaming multicast). As expected.

[jordens]

This is likely then a W5500 or w5500 specific bug.

[ryan-summers]

Do you have a router and/or managed switch in between? The switch I'm using with my local network is just some cheap unmanaged unit. I suspect that it's forwarding the multicast traffic to all of the ports regardless of subscription to the multicast group.

[jordens]

There is a cisco 2960L in between. Whether that one can be called "managed" or not is debatable. But it does behave properly regarding non-flooding of multicast.

[aferk]

We have found a similar issue when using the booster with an unmanaged switch with other traffic. The booster seems to reboot when it receives invalid packages on the MQTT port 1883. After that, when trying to enable channels, they only seem to change to powered instead of enabled and stay this way until the booster is powercycled.
This behavior can be reproduced by sending invalid packages via netcat, e.g. nc -u <booster-ip> 1883 < /dev/random.

We will use a vlan for now, but it would be nice if this could be fixed to make integration into existing network structures easier.

[jordens]

@aferk Contributions welcome.

[ryan-summers]

@aferk if you connect to the device when it's in this state and run the platform service command and attach the output here, that would be helpful.

That will tell us why the device is resetting.

[aferk]

Running service before sending packages:

> service
Version             : v0.5.0 [debug]
Hardware Revision   : v1.6
Rustc Version       : rustc 1.76.0 (07dca489a 2024-02-04)
Features            : 
Detected Phy        : W5500
Panic Info          : None
Watchdog Detected   : false

Sometimes I just get the Watchdog Detected: true as above, but when re-trying a few times, I do also get the more informative panic info below:

> service
Version             : v0.5.0 [debug]
Hardware Revision   : v1.6
Rustc Version       : rustc 1.76.0 (07dca489a 2024-02-04)
Features            : 
Detected Phy        : W5500
Panic Info          : panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/w5500-0.4.1/src/raw_device.rs:92:13:
attempt to subtract with overflow

Watchdog Detected   : true

Booster HW is version 1.6 (non-HL) from Creotec. We tried to upgrade the firmware because of this issue sinara-hw/Booster#393, which both of our non-HL boosters regularly experience.

[jordens]

@aferk The old firmware is dead and rotten. You will hopefully understand that this firmware here is bound to have the same fate as long as people don't invest into it or contribute or buy where the development is funded.

[ryan-summers]

The panic referenced above should be fixed in the latest release of the w5500 crate, which was updated recently (in #373). I'm currently testing a fix for the watchdog event in smoltcp