Add camera tracking module #1469
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
srmainwaring
force-pushed
the
prs/pr-camera-tracking
branch
from
3b85ced
to
9119a92
Compare
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
- Initial structure for camera tracking module - Add initial structure for module - Use MPImage to render RTSP stream - Subclass MPImage to allow tracker overrides - Abstract an interface for the tracker and provide dlib example - Enable title - Add cv2 CSRT tracker - Document TODOs - Rename image to tracker_image - Add tracker to forward mavlink commands - Add dev notes - Add requests for camera and gimbal info - Preparation for sending mavlink camera track commands - Add event class and enums - Remove event class and simplify - Note: mp_image has a different event model to param_edit etc. - Add mavlink cmds to start / stop tracking - Capture command_acks - Add variables for camera and gimbal sysid and compid - Add initial onboard controller script - Remove print of command_long - Add handlers to onboard controller - Format ack messages - Format debug messages - Onboard controller: send camera info - Onboard controller: use enum for flags - Add onboard controller and gimbal controller classes - Fix missed lock -> _lock rename - camera_view: remove hardcoded tracking coords - onboard controller: add video capture and tracker - camera_view: disable repeated discovery requests - onboard controller: update controller gains - camera_view: add example RTSP URLs - camera_view: suppress mtype checks - onboard controller: thread safe handling of mavlink messages - camera_view: fix handling of image resolution when setting track rectangle - onboard controller: suppress profiling output - camera_view: additional mavlink debug info - onboard controller: additional mavlink debug info - onboard controller: add command line options - camera_view: add command line options - onboard controller: add usage docs - onboard controller: initial support to send tracking image status - camera_view: enable one time discovery requests - camera_view: use MAV_CMD_SET_MESSAGE_INTERVAL to request image status - onboard controller: populate image status using initial target - camera_view: tidy module docstring Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
…in sub-classes Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
- tracker_image: override proces_event to handle update to tracked rectangle - onboard controller: image status sends current tracked rectangle - camera_view: update mavlink handlers - camera_view: remove unused code - camera_view: handle tracking image status - tracker_image: override proces_event to handle update to tracked rectangle (fix) - camera_view: request image status at 20Hz - camera_view: remove unused code - camera_view: remove unused code - camera_view: suppress debug prints - tracker_image: remove unused code - onboard_controller: remove unused code - onboard_controller: suppress debug prints - tracker_image: update docstring - tracker_image: add set_position method to Tracker base class - tracker_image: add set_position method to TrackerCSTR - tracker_image: set position of tracked rectangle - onboard_controller: clean up track request print statements - camtrack: onboard_controller: add timeout and restart Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
- camera_view: add variable for tracking image status update rate - camera_view: camera and gimbal component ids should match autopilot Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
- onboard_controller: request gimbal device attitude status - onboard_controller: send image status at 20 Hz - onboard_controller: adjust default controller gains Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
srmainwaring
force-pushed
the
prs/pr-camera-tracking
branch
from
c24fa2f
to
01f5846
Compare
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
- Intial tune of status rates and pid gains - Reduce image status rate to 20 Hz - Set response_target in set message interval to flight-stack default - Add debug info for image status / tracker_pos - onboard_controller: apply formatter - onboard_controller: adjust controller gains - Port basic PID controller from ArduPilot AC_PID - onboard_controller: replace PID controllers - onboard_controller: use normalised coords for tracking - onboard_controller: increase k_i for tracking controllers - onboard_controller: add optional live graphs for pid analysis - onboard_controller: close connection on exit - Add settings and defaults Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
srmainwaring
force-pushed
the
prs/pr-camera-tracking
branch
from
04e97bb
to
feab200
Compare
- onboard_controller: move pid_basic to lib - Prevent transitive import of dlib and wx - onboard_controller: check connection valid before close - onboard_controller: add tracker option Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
srmainwaring
force-pushed
the
prs/pr-camera-tracking
branch
from
16edd5d
to
fd05617
Compare
… rate Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
- Add missing import Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
… RPi Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Signed-off-by: Rhys Mainwaring <rhys.mainwaring@me.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a camera tracking module to support external camera tracking where the object tracker and gimbal controller run on an onboard companion computer.
The module is intended to work with Asif Khan's (@khanasif786) 2024 GSoC Visual Follow Me project and the modules make extensive use of the Python examples developed as part of the project.
Status
Related Blogs and PRs
Details
There are two parts to the package: 1. a MAVProxy module
camtrackthat provides a GUI showing a RTSP video stream that allows a target to be selected, and 2. a Python script intended to run on a companion computer that receives camera tracking commands relayed from the flight-controller, attempts to track the object using a configurable tracking class, then updates the gimbal pitch and yaw rates.The GUI module reuses the
mp_imagemodule by subclassing from the objects in that module so the tracking methods can be overridden. This is required to allow the tracking logic to be moved out of the GUI and into a separate moduleonboard_controller.pythat may run either on a PC (in the case of simulation) or on a companion computer (in the case of a real vehicle).The module will work with both simulated (Gazebo/SITL) and real vehicles and is tested with both.
Testing - Gazebo
A modified version of the
omnirover_playpen.sdfworld in ArduPilot/SITL_Models contains objects of interest and two models equipped with gimbal mounted cameras.Figure: Gazebo
omnirover_playpenworld for testing camera trackingEnable the UDP stream
gz topic -t /world/playpen/model/iris_with_gimbal/model/gimbal/link/pitch_link/sensor/camera/image/enable_streaming -m gz.msgs.Boolean -p "data: 1"Convert to a RTSP stream using the scripts in
ardupilot/libraries/AP_Camera/examples:Launch a SITL session for the iris quad-copter (DDS support is optional):
Load the module:
STABILIZE> module load camtrackRun the onboard controller script:
Shift + Left-Mouse-Clickto select a target.Ctlr + Left-Mouse-Clickto cancel targeting.Figure: MAVProxy
camtrackmodule GUIFigure:
onboard_controller.pyterminal outputFigure: MAVProxy console output
