Multiple AI deck streaming

[Ben-Jarvis]

I'm trying to stream images from multiple AI decks while flying and find that when trying to stream from multiple Crazyflies at the same time, one by one they get unstable and crash.

I've configured the crazyflies to connect to my network and they can stream video stably at ~7 fps. I can also fly crazyflies normally without the aideck attached. However, when I attach the aideck they will get unstable, seemingly at random, fail the dynamic check and then crash. Typically one will crash every 10-20 seconds, but the timing varies. Other than the aideck, the only deck attached to the crazyflies is a single mocap marker.

With 4 crazyflies + aidecks I've tried 1-4 crazyradios (single channel per radio) but it hasn't made a difference. i.e one radio/channel for all crazyflies, a separate radio and channel for each, or 2 pairs of each. I've also tried different channels from 20-120, when using multiple channels they're separated by at least 4. Always at 2MHz bandwidth, and I've found radios disconnect due to packet loss even at 1MHz. Normally I am logging Odometry, but I've tried without logging and the issue still occurs.

I was wondering if there are other suggestions for what I could try, or if you think this is even possible?

Let me know if you need anymore info.

Setup:
Ubuntu 22.04
ROS2 Humble
Motive 2.3.1
Crazyflie 2.1
Crazyradio PA

[whoenig]

Which backend do you use? We have flown 3-4 CFs with AI deck using the cpp backend before. There is definitely some interference between radio and Wifi and things get worse when the motors are one (i.e., if you are flying).

If you use the latest version and the cpp backend, there are some very handy communication statistics in the //status topic and (if enabled) in the //connection_statistics topic. It would be helpful to post these. For testing, it also helps to simply remove the propellers for testing to avoid actual crashes and watch out on the topics mentioned above for strange changes whenever you enable the wifi streaming.

[Ben-Jarvis]

Thanks for getting back so quickly! I'm using the cflib backend, because I've been using the cmd_hover topic for teleop. I can switch to cpp and fullstate, but for now I've just done a hover test with the cpp backend.

Doing the cpp hover they did last longer before crashing compared to the cflib, but I still saw similar behaviour. Here's images of the data from the //status topics for each of the 4 crazyflies that were flying. They were all streaming at ~7fps for the whole test, even after crashing (for the two that did).

[whoenig]

It looks like you are running an older STM32 firmware, so we can't see any rx numbers (they are 0)? You need bitcraze/crazyflie-firmware@0bacd86 (either via using the master firmware, or you cherry-pick this change if you use a custom firmware).

What is your mocap frequency? It shouldn't be more than 100 Hz.

[Ben-Jarvis]

Hi @whoenig, so sorry to be getting back to you so late on this! I'm still having the same issue with some crazyflies getting unstable and crashing when I'm trying to stream multiple ai-decks and fly. So would love if you could spot what I'm doing wrong or give me other things to check. Here is some data from a flight of 4 CFs, all with AI-decks, and flying the infinite_flight example with the cpp backend. One of the CFs crashed immediately after takeoff (not shown) then you can see 'cf06' crashed at approx 70 seconds. I also only started the image streaming on my pc at around 150 seconds when the unicast rate (tx and rx) drops and the rx_broadcast drops.

Altitude and unicast latency:

Communication statistics:

I'm using Motive 2.3.1, with the following settings:

Additionally, I get alot of high latency and low unicast rate warnings during runtime:

Any help would be really appreciated!

[Ben-Jarvis]

Hi @whoenig, just wanted to follow up to let you know that I also experience crashes at random intervals even when flying just a single crazyflie with the AIdeck (without even starting the image streaming). During these flights the communication seems fine with minimal dropped packets. Though there is often a latency spike about a second before crashing, though not larger than spikes I've seen during normal flight when the drone doesn't crash. Also as in the attached picture this didn't occur and the drone still crashed.

Just wondering if this is behaviour you've observed as well, or if you think there is something I'm doing wrong. Thanks!

Also, when I do stream my resolution is 324x244 and I'm using JPEG encoding, though that's not relevant to this case.

[whoenig]

All the data you show look reasonable to me. Even the warnings you get may be "normal", since the thresholds are pretty low by default. Latencies below 30 and rates above 0.7 shouldn't cause major issues.

Perhaps a better way would be to figure out the cause of the crash directly. Once it crashes, does it reboot and the LED blinks fast? If so, you can connect via the CF client and click in the console tab on "Assert Info" to get some information why it rebooted. If not, it would be useful to also add the uSD-card deck (this requires some soldering and a custom firmware, but I believe Bitcraze has documented the process) and log data on-board at a higher rate, e.g., to see when and what kind of motion capture information was received.