Diagnosing Communication and Mocap issues

[Eric-Ewing]

We're having problems with some fraction of messages/commands not being received by crazyflies.

We've had these problems with high level commands, some low percentage of the time crazyflies will not execute a goto/takeoff/land command that was sent. I've seen it happen with as few as one crazyflie, but happens more with larger numbers of drones (unclear if at a higher rate as well, or just because there are more opportunities to miss a command).

An example can be seen in this video: https://youtu.be/mFtb2K1SdCE.
The purple (pink?) crazyflie should have been the first to move, it was sent a goTo command, but didn't react to it and then made a somewhat miraculous recovery as we switched to frequent cmdPositions. (I've run it other times successfully, so it's not an issue with the python script).

To look for a root cause (and avoid crash anything), I made a script that changes the colors of the each crazyflie once a second (looping through each crazyflie and calling cf.setLEDColor), rotating between red, green, and blue. With the mocap data, some percent of LED param changes don't go through and result in the LED being some color other than full red, green, or blue. Here's what it looks like with mocap data streaming:

with_mocap.mp4

However, if we pause motion capture, the commands go through as normal. (Ignore anything that doesn't change in this one, it didn't properly connect in the beginning for other reasons or doesn't have an LED ring. I fixed them after I took the videos and the results didn't change)

without_mocap.mp4

These experiments were with ~27 crazyflies across 2 radios, but when I run the same experiment with fewer crazyflies (e.g. 5) the results are very similar. And again, I've seen a single crazyflie miss a takeoff command.

We've just set up a new set of Vicon Valkyrie cameras, which have 150fps natively. I've reduced the framerate down to 50 and 30 (the lowest Tracker will allow) and see no noticeable change in reliability of colors. It seems related to motion capture, because the problem goes away if we pause the motion capture data, but it doesn't seem tied to the framerate as far as I can tell. Reducing the framerate from 150 to 30 made no noticeable difference. When I reduce framerate, I do it in Tracker and update the qos param in the mocap yaml file. I've double checked that the poses are getting updated at the appropriate hz as well. Is there anything that I'm missing?

Broadcast commands work great as far as I can tell. We have hovered ~30 crazyflies with a variation of nice_hover.py with a mocap fps of 150 at the time, I believe.

Any ideas what the problem could be or other experiments to run?

[whoenig]

I have seen these issues as well (both Crazyswarm1 and 2). I think the root cause is in the firmware and/or https://github.com/bitcraze/crazyflie-link-cpp, because there is no end-to-end flow control. In other words, there are places in the firmware that might drop a radio packet after it has been acknowledged due to queue size limitations. This would also explain, why the problem is more apparent when mocap is enabled (which sends lots of broadcast commands).

To fix it, we need some more awareness at Bitcraze (@ataffanel) and perhaps a test application that reproduces the issue without the need for Crazyswarm2 (just crazyflie-link-cpp, since the official Python lib doesn't support broadcasts, yet).

As workaround, I sometimes send "critical" commands multiple times in a loop. For broadcasts we do that automatically within Crazyswarm2 (since there is no ack), but for unicast we don't.