Bitcraze
Neuromorphic (Dynamic Vision) Sensing: Crazyflie meets Hoverfly! #801
Replies: 14 comments 13 replies
-
|
Hi, I'm happy that you find that the Crazyflie is a useful platform, that is what we are hoping and aiming for! Keep us posted on your progress, I'd love to hear what is happening in the future! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @krichardsson! As you probably know, the event cameras (DVS) have a huge bandwdith (~10^5 events per second), requiring a high-speed USB or UART connection. Thankfully the Teensy line of MCUs supports this, and with Teensy 4.0 I can also use the fast CPU (600MHz) to do essential pre-proccessing on the event stream. Although the 128x128 retina of the DVS would support all kinds of cool algorithms (e.g., optical flow for tunnel navigation), my current goal is just to have the Teensy compute a simple 2D vector (velocityY, velocityZ) which will get passed along to the Crazyflie's state estimator. Having found the Teensy's UART to be more robust than I^2C for this kind of work, I'm using another of the Teensy UARTs to send the data to the Crazyflie. (Thanks to your excellent new-deck tutorial, I can even monitor the communications wirelessly in the GUI client!) Currently I'm just sending a stream of alphabet characters to validate the comms, but obviously I want to send those two velocity values, probably as 16-bit signed ints. Normally I'd use MSP for this (which I see is already built into the firmware), but looking at the deck drivers I don't see MSP being used for them. So my current plan is to borrow some ideas from your GTGPS driver code. But if you think a different approach might be better, please weigh in! |
Beta Was this translation helpful? Give feedback.
-
|
I think your approach looks sound, pre-processing on the deck sounds like a good idea :-) Just sending the essential information to the Crazyflie STM reduces the load, processing/filtering the data in the deck (if possible) will be more efficient than transferring it to the Crazyflie and filtering it there. Just getting the data out of the UART will add some load, so minimizing the amount of data to transfer can be beneficial. You can also look at the lighthouse deck for inspiration, we use a UART to transport data from the deck to the Crazyflie. In that case (with 4 base stations) we push 1600 frames (of 12 bytes) per second to the Crazyflie. I'm not exactly sure, but I assume |
Beta Was this translation helpful? Give feedback.
-
|
Cool, thanks @krichardsson! MSP = Multwii Serial Protocol. The lighthouse deck code looks like what I'm currently doing myself. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, that explains it! I found some definition on the internet that could have been something like a packet based protocol :-) |
Beta Was this translation helpful? Give feedback.
-
|
My first Bolt 1.1 flight! |
Beta Was this translation helpful? Give feedback.
-
|
sorry for the delay! We are in and out of our summer holidays. That is looking good! |
Beta Was this translation helpful? Give feedback.
-
|
LOL, no apologies necessary -- hope you enjoyed the well-deserved break! I'm pretty close to getting DVS-based altitude-hold working in the Webots Crazyflie sim. As usual, a well-tuned EKF is key. Here's a comparison between the ground-truth altitude/variometer and the corresponding EKF-filtered values from the simulated DVS: |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Nice! |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Lateral hold based on simulated DVS works even better than altitude-hold! When I hit the roll stick pretty hard, it still manages to stabilize. |
Beta Was this translation helpful? Give feedback.
-
|
that looks very good! soon on the real drone as I can see 😄 |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Looking good! |
Beta Was this translation helpful? Give feedback.
-
|
cool! |
Beta Was this translation helpful? Give feedback.
-
|
LOL, if I had bothered to RTFM, I would've realized that the Bolt has its own onboard voltage regulator, so if I use a 2s lipo or larger, I don't even need that external regulator on my design! |
Beta Was this translation helpful? Give feedback.
-
The DC/DC has a 500mA limit (where ~100mA is used by the Crazyflie) to take into account. Just a heads up. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the heads-up, Tobias. Based on the prototype in the photo above, it appears that the DC/DC will support enough current draw to power the Teensy, TinyPICO, and DVS. But I appreciate that this may end up being too much current draw, in which case I'll have to look into a separate powering configuration. |
Beta Was this translation helpful? Give feedback.
-
|
A working deck prototype! |
Beta Was this translation helpful? Give feedback.
-
|
That's awesome! Now to see if it flies 😄 |
Beta Was this translation helpful? Give feedback.
-
|
Well done! |
Beta Was this translation helpful? Give feedback.
-
|
Very nice, great work! What si the total mass of the deck and the DVS, including the lense? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, @matejkarasek! The prototype weighs in at 12.7g, but the new version will probably be a gram or two more (slightly larger, with step-up voltage regulator to power the boards from the battery instead of Bolt VCC. |
Beta Was this translation helpful? Give feedback.
-
|
Sweet, too bad the mini eDVS is discontinued... |
Beta Was this translation helpful? Give feedback.
-
As I was discussing in my posts about simulation in UnrealEngine, my current research program involves flight control using biologically-inspired ("neuromorphic") sensors like the iniVation Dynamic Vision Sensor. I just started my first deck driver for the eDVS and am looking forward to getting it talking to the Crazyflie. While waiting for my deck PCB to arrive, I've been working on a simulation of the DVS. As shown in this video, you can simulate DVS events by first-differencing (subtraction) of two successive images captured by the simulated camera. Thanks to the work of @knmcguire on the Python API for Webots, it was easy to modify the wall-following example to work with the simulated DVS. I have an undergraduate student starting with me on Monday whose first job will be to see whether the simulated DVS events can be used for further stabilization of the vehicle, as used by flying insects in lieu of an accelerometer / altimeter. Once we have both working (simulator and deck) I hope to do a blog post on this.
Among many things that excite me about Crazyflie is the way you've incorporated the Arduino-style calls into the Deck API. I spent several years building my own flight-control package around the Arduino API for this kind of research, but switched to Crazyflie recently when I learned about the Bolt 1.1 brushless board (because the sensors I'm working with need more lift than you get with a brushed build). I figured it was worth giving up the Arduino-style coding in order to get all the features of Crazyflie (esp. hover-in-place and swarms), so I'm thrilled to see that with Crazyflie I can apparently have the best of both worlds!
Unfortunately, iniVation has discontinued the tiny, five-gram Mini eDVS, but there are lots more opportunities for neuromorphic research with Crazyflie. For example, the latest generation of tiny FPGAs may make it possible to do onboard modeling of Spiking Neural Networks (SNNs), as described here.
Beta Was this translation helpful? Give feedback.
All reactions