Feature request: Local image and video recording (MAVLink camera control)

[ddd999]

Real-time video streaming is great, but for any sort of photography, videography, or photogrammetry mission, recording full-resolution still images (or video) is an absolute necessity.

It's obviously not practical to transfer that much data over the wifi link, so the solution is to have rpanion respond to MAVLink camera control commands from the flight controller or ground control station and storage the imagery data on the local SD card (or whatever storage is available to rpanion).

I would love to implement this myself if I had the skills, but a read through the code tells me I do not (yet?). Conceptually it seems relatively simple. MAVLink has an existing set of commands for this purpose, and libcamera can be used for capturing camera data and adjusting camera settings (exposure, focus, zoom, etc.).

[stephendade]

That would be a nice feature to have! If you want to implement it, I'd be happy to provide advice :)

The ./server/ntrip.js module would serve as a good starting point, as it takes in MAVLink packets and performs some actions based on them.

[ddd999]

I will give it a go and see how far I can get!

[ddd999]

So I have made some progress setting this up. With this commit: ddd999@2cba554) I am able to detect various different MAVLink messages with onMavPacket in ./server/camera.js.

However, Mission Planner (ArduPilot GCS software) sends COMMAND_LONG for a lot of control messages (including the camera trigger), and for some reason I can't detect COMMAND_LONG, or the COMMAND_ACK response from the flight controller within Rpanion. Using the MAVLink Inspector in Mission Planner, I can confirm that these commands are indeed being sent and ACK'd.

I know that Rpanion is able to successfully send a COMMAND_LONG, because the "reboot flight controller" button works.

Any idea what I'm missing here?

Edit: I see the issue. COMMAND_LONG is a point-to-point message so I have to intercept that before it gets to the vehicle. Looks like I also need to run some simulations to see what ArduPilot does on an autonomous waypoint mission.

[stephendade]

Edit: I see the issue. COMMAND_LONG is a point-to-point message so I have to intercept that before it gets to the vehicle. Looks like I also need to run some simulations to see what ArduPilot does on an autonomous waypoint mission.

Yep, correct. The correct method would be to send the camera trigger to the Rpanion's sysid/compid (255,1), not ArduPilot's.

There's some MAVLink routing basics in https://mavlink.io/en/guide/routing.html

There'd need to be a few supporting changes:

• Configurable component ID for Rpanion. The sysid it can just grab from the connected flight controller.
• Rpanion needs to send out heartbeat packets, so any connected GCS software can detect it as a separate entity

Then, the user will use the GCS to select Rpanion (via sysid/compid) and send the camera trigger command.

[ddd999]

Hmm, I am definitely in over my head here. If I understand correctly, Rpanion needs to advertise itself as a component of the vehicle of type MAV_TYPE_CAMERA, in order to accept camera control commands.

I'm just not sure how to get there. I guess the mavManager class is where the changes would need to happen? In order for this to work does Rpanion actually need to have to instances of mavManager running-- one for talking to the flight controller, and one for accepting/responding to camera control commands? Or can one instance of the class handle everything? I'm not sure if this is the correct venue for this discussion so let me know if there is a better place.

[ddd999]

Oops, clicked the close button by accident!

[stephendade]

Hmm, I am definitely in over my head here. If I understand correctly, Rpanion needs to advertise itself as a component of the vehicle of type MAV_TYPE_CAMERA, in order to accept camera control commands.

Quite possibly. I'd need to investigate how the GCS's handle this. I'd prefer a more generic MAV_TYPE_ONBOARD_CONTROLLER, since not all users are using cameras. Maybe we can have an option for "Send MAVLink information" in the UI, which will trigger the MAV_TYPE_CAMERA heartbeats to be sent.

In order for this to work does Rpanion actually need to have to instances of mavManager running-- one for talking to the flight controller, and one for accepting/responding to camera control commands? Or can one instance of the class handle everything?

One instance can handle this fine. We'd need to add the appropriate functions into ./mavlink/mavManager.js

Note this has a few linkages to #169.

I can add the heartbeat and GUI option sometime this week.

[ddd999]

Beat you to it :) Please review PR #170 when you have time, thanks!

[ddd999]

I'm (very very) slowly trying to work my way up to adding this capability, but thought I should check in on the overall approach / conceptual design first.

Here's what I'm thinking:

Frontend
A new top-level page on the UI for MAVLink Camera Control:

• Enable/Disable button
• Status display (i.e. Active / Not Active)
• Form inputs for camera capabilities (to respond to MAVLink queries with CAMERA_INFORMATION and CAMERA_CAP_FLAGS). Perhaps these could default to the Raspberry Pi camera. Future: allow selection/saving of presets, or auto-detect capabilities where possible
• Form input for image/video storage path

Backend
As I understand it, this is how it should work:

• camera.js re-uses the mavManager instance that gets created in flightController.js for sending MAVLink messages (making this work is eluding my limited knowledge of JS as yet...)
• camera.js has a 1s interval function for sending out camera-specific MAVLink heartbeats
• camera.js has its own onMavPacket() function, which responds to MAVLink messages targeting the camera component
• camera.js: additional functions for interfacing with libcamera to do the actual image/video capturing, adjusting camera settings, etc.

Let me know if this makes sense, or if there are better ways to handle this. If I understand the code correctly, mavManager.js may need some changes to be able to process MAVLink Command Protocol messages (COMMAND_INT and COMMAND_LONG).

[stephendade]

I suspect you're wanting the same as in #169?

I'd prefer to have this sort of control as part of the current video.js and videostream.js, so that there's a single point for all video streaming functions.

Something like an "Allow MAVLink control" checkbox in video.js. This would enable to camera-specific heartbeat and MAVLink handling. There'd be sub-options for the camera/image storage path.

EDIT:
Another reason against using a separate page/function would be managing conflicts - if the same camera device was selected in both pages.

[ddd999]

That makes total sense to include it as part of the Video page. It would definitely be best to integrate these features as much as possible.

Yes, there would definitely be a lot of similarities with #169.

[ddd999]

I'm pondering more about how to implement this, I think by starting with still-image capture. That's (maybe) simpler, but also is the feature that I'm most interested in as it would enable photogrammetry missions.

Ideally, it would be possible to maintain a live video stream and capture full-resolution, images at the same time. This would, require interrupting the video stream briefly to capture a still, since as far as I know most cameras can only do one or the other, not both simultaneously. (Edit: in the very distant future, it might be useful to have multiple camera support so one can stream low-bitrate video while the other captures stills/high-bitrate video)

Here's my thought for how to achieve local still-image recording:

Frontend:

• Add controls to the Video page to retrieve and configure still camera settings (resolution, white balance, autofocus, etc.). My main concern here is that the Video page would become too cluttered as this would at least double the number of controls. Maybe this needs its own top level page?
• Add a checkbox option to allow simultaneous video streaming, with a warning that the stream would be interrupted whenever a still is being captured.

Backend:

• Add a function that responds to MAV_CMD_DO_DIGICAM_CONTROL messages. In the (distant) future, this could be extended to implement PARAM_EXT_XXX messages: https://mavlink.io/en/services/camera_def.html). I think this might require sending another heartbeat to advertise a MAVLink camera device?
• Add a function that retrieves still camera capabilities
• Add a function to capture a still image using the settings that were selected in the frontend
• When a still image capture is requested, stop the video stream, record the still, and restart the video stream with the previous settings. If I understand correctly, this could be handled by the existing startStopStreaming function, but maybe there is a more efficient way to do this (e.g., in a new "pauseStreaming" function)?

After writing this out I'm realizing how big of a task this will be, so any feedback or advice would be greatly appreciated.