Vision-Based-Control

code for "Adaptive Vision-Based Control of Redundant Robots with Null-Space Impedance for Human-Robot Collaboration", ICRA 2022

If you find our method useful in your research, please consider citing:

@INPROCEEDINGS{Yan2022Adaptive,
  author={Yan, Xiangjie and Chen, Chen and Li, Xiang},
  booktitle={2022 International Conference on Robotics and Automation (ICRA)}, 
  title={Adaptive Vision-Based Control of Redundant Robots with Null-Space Interaction for Human-Robot Collaboration}, 
  year={2022},
  pages={2803-2809},
  doi={10.1109/ICRA46639.2022.9812218}}

Prerequisites

• Ubuntu 18.04
• ROS Melodic
• Unity 2020.3.12f1c1 (2020.3.8f1 or later shall also work)
• Unity-Robotics-Hub
• Mixed Reality Toolkit
  • Mixed Reality Toolkit Fondation 2.7.2
  • Mixed Reality Toolkit Standard Assets 2.7.2
  • Mixed Reality OpenXR Plugin 1.0.2
• Hololens 2 Emulator

For real experiments:

• Balser camera driver acA1440-220uc
• UR5
• Hololens 2

Installation

Install the following dependences in your python3 env:

pip install numpy
pip install matplotlib
pip install rospkg
pip install PyYAML
pip install urdf-parser-py
pip install pykdl_utils

Then, clone the repo:

git clone

Build the catkin workspaces:

cd <YOUR_PATH>/Vision-Based-Control/ws_icra2022
catkin_make

HoloLens 2 Deployment

Please refer to this readme file.

Packages

all the packages are in /Vision-Based-Control/ws_icra2022/src

• aruco_ros: for aruco. cloned from pal-robotics/aruco_ros, no modification.
• dragandbot_common: a driver for pylon camera, cloned from dragandbot/dragandbot_common, no modification.
• pylon-ros-camera: a driver for pylon camera cloned from basler/pylon-ros-camera, where i modified default.yaml for calibration according to the README in it.
• fmauch_universal_robot and Universal_Robots_ROS_Driver: drivers for UR5, cloned from UniversalRobots/Universal_Robots_ROS_Driver, no modification. ----submodule
• easy_handeye: for eye hand calibration. i modified ur5_basler_calibration.launch
• ROS-TCP-Endpoint: for communication between Unity and ROS
• my_pkg: main control code
• tcst_pkg: control code we refered to. from Musyue/tcst_pkg. ----submodule

Usage

cd ws_icra2022
source devel/setup.bash
roslaunch my_pkg ur5_bringup.launch # for UR5
roslaunch ros_tcp_endpoint endpoint.launch # for hololens 2
roslaunch my_pkg ibvs_run_eye2hand.launch # for control