Skip to content

A Survey of Embodied Learning for Object-Centric Robotic Manipulation

License

Notifications You must be signed in to change notification settings

RayYoh/OCRM_survey

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

22 Commits
 
 
 
 
 
 

Repository files navigation

Awesome Object-Centric Robotic Manipulation

Arxiv Awesome license Maintenance PR's Welcome

overview

This is the repository of A Survey of Embodied Learning for Object-Centric Robotic Manipulation, a comprehensive review of latest advancements in object-centric robotic manipulation including embodied perceptual learning, embodied policy learning, and embodied task-oriented learning. For details, please refer to:

Any contributions to this list are most welcome! Please feel free to open issues or create pull requests as follows:
  • Fork the project into your own repository.
  • Add the Title, Paper link, Venue, Code/Project link in README.md using the following format:
    |[Title](Paper Link)|Venue|[Code/Project](Code/Project link)|
    
  • Submit the pull request to this branch.

🔥 News

Last update on 2024/08/28

🔨 Abstract

Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI. It is crucial for advancing next-generation intelligent robots and has garnered significant interest recently. Unlike data-driven machine learning methods, embodied learning focuses on robot learning through physical interaction with the environment and perceptual feedback, making it especially suitable for robotic manipulation. In this paper, we provide a comprehensive survey of the latest advancements in this field and categorize the existing work into three main branches: 1) Embodied perceptual learning, which aims to predict object pose and affordance through various data representations; 2) Embodied policy learning, which focuses on generating optimal robotic decisions using methods such as reinforcement learning and imitation learning; 3) Embodied task-oriented learning, designed to optimize the robot's performance based on the characteristics of different tasks in object grasping and manipulation. In addition, we offer an overview and discussion of public datasets, evaluation metrics, representative applications, current challenges, and potential future research directions.

đź“ť Citation

@article{zheng2024ocrm,
  author= {Zheng, Ying and Yao, Lei and Su, Yuejiao and Zhang, Yi and Wang, Yi and Zhao, Sicheng and Zhang, Yiyi and Chau, Lap-Pui},
  title= {A Survey of Embodied Learning for Object-Centric Robotic Manipulation},
  journal={arXiv preprint arXiv:2408.11537},
  year={2024}
}

Embodied Perceptual Learning

Data Representation

Image-Based Representation

Paper Venue Code/Project
Real-time seamless single shot 6d object pose prediction CVPR 2018 -
MonoGraspNet: 6-DoF Grasping with a Single RGB Image ICRA 2023 -
RGBGrasp: Image-based Object Grasping by Capturing Multiple Views during Robot Arm Movement with Neural Radiance Fields RAL 2024 Project
RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation ICRA 2024 Project
Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects CoRL 2022 Project
Clear-Splatting: Learning Residual Gaussian Splats for Transparent Object Manipulation ICRAW 2024 -

3D-Aware Representation

Depth-based representations

Paper Venue Code/Project
RGB-D object detection and semantic segmentation for autonomous manipulation in clutter IJRR 2018 -

Point cloud-based representations

Paper Venue Code/Project
Shape completion enabled robotic grasping IROS 2017 -
PointNetGPD: Detecting Grasp Configurations from Point Sets ICRA 2019 Project
3D Implicit Transporter for Temporally Consistent Keypoint Discovery ICCV 2023 Code

Other representations

Paper Venue Code/Project
RVT: Robotic View Transformer for 3D Object Manipulation CoRL 2023 Project
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields CoRL 2023 Project
ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation ECCV 2024 Project
Object-Aware Gaussian Splatting for Robotic Manipulation ICRAW 2024 Project

Tactile-Based Representation

Paper Venue Code/Project
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force Sensors 2017 -
Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation RAL 2020 Project
Allsight: A low-cost and high-resolution round tactile sensor with zero-shot learning capability RAL 2023 Code
Learning to grasp without seeing ISER 2018 Project
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks TRO 2020 Project
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes? CoRL 2017 -
Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play CoRL 2023 Project
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch RAL 2018 -
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations CoRL 2022 Project
TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors CoRL 2022 Code
Efficient Tactile Simulation with Differentiability for Robotic Manipulation CoRL 2023 Project
Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation CoRL 2023 -
Tactile-Augmented Radiance Fields CVPR 2024 Project

Object Pose Estimation

Instance-Level Object Pose Estimation

Paper Venue Code/Project
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes Introduction RSS 2018 Project
CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation ICCV 2019 -
6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation CVPR 2024 -
Match Normalization: Learning-Based Point Cloud Registration for 6D Object Pose Estimation in the Real World TPAMI 2024 -
Deep fusion transformer network with weighted vector-wise keypoints voting for robust 6d object pose estimation ICCV 2023 Code

Category-Level Object Pose Estimation

Paper Venue Code/Project
Normalized object coordinate space for category-level 6d object pose and size estimation CVPR 2019 Project
Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation ICCV 2021 Project
Rigidity-aware detection for 6d object pose estimation CVPR 2023 -
Revisiting Fully Convolutional Geometric Features for Object 6D Pose Estimation ICCV 2023 Code
Category-Level Articulated Object Pose Estimation CVPR 2020 Project
Towards Real-World Category-level Articulation Pose Estimation TIP 2022 -
CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation ICRA 2022 Project
ShAPO: Implicit Representations for Multi Object Shape Appearance and Pose Optimization ECCV 2022 Project
FSD: Fast Self-Supervised Single RGB-D to Categorical 3D Objects ICRA 2024 Project
DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation IROS 2024 Project

Novel Object Pose Estimation

Paper Venue Code/Project
Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images ECCV 2022 Project
GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence CVPR 2024 Code
OnePose: One-Shot Object Pose Estimation without CAD Models CVPR 2022 Project
Zero-Shot Category-Level Object Pose Estimation ECCV 2022 Code
SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation CVPR 2024 Code
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects CVPR 2024 Project

Affordance Learning

Affordance Learning by Supervised Learning

Paper Venue Code/Project
Affordancenet: An end-to-end deep learning approach for object affordance detection ICRA 2018 Code
Grounded Human-Object Interaction Hotspots From Video ICCV 2019 Porject
Affordances from Human Videos as a Versatile Representation for Robotics CVPR 2023 Project
Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation ECCV 2024 Porject
RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation arXiv 2024 Project
Open-Vocabulary Affordance Detection in 3D Point Clouds IROS 2023 Project
Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding arXiv 2024 Code

Affordance Learning From Interaction

Paper Venue Code/Project
Where2Act: From Pixels to Actions for Articulated 3D Objects ICCV 2021 Project
AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions ECCV 2022 Porject
DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation ICLR 2023 Project
Self-Supervised Learning of Action Affordances as Interaction Modes ICRA 2023 Porject
Information-driven Affordance Discovery for Efficient Robotic Manipulation arXiv 2024 Project
Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects NeurIPS 2023 Project
Articulated Object Manipulation with Coarse-to-fine Affordance for Mitigating the Effect of Point Cloud Noise ICRA 2024 -
Learning environment-aware affordance for 3d articulated object manipulation under occlusions NeurIPS 2023 Code
End-to-End Affordance Learning for Robotic Manipulation ICRA 2023 Project

Embodied Policy Learning

Policy Representation

Explicit Policy

Paper Venue Code/Project
Robotic grasping using deep reinforcement learning CASE 2020 -
Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation CoRL 2018 Porject
Rapid Motor Adaptation for Robotic Manipulator Arms CVPR 2024 -
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot IROS 2023 Code
Learning insertion primitives with discrete-continuous hybrid action space for robotic assembly tasks ICRA 2022 Project

Implicit Policy

Paper Venue Code/Project
Regularizing Model-Based Planning with Energy-Based Models CoRL 2019 -
Implicit Behavioral Cloning CoRL 2021 -
Implicit Distributional Reinforcement Learning NeurIPS 2020 Code
Energy-Based Imitation Learning AAMAS 2021 Code

Diffusion Policy

Paper Venue Code/Project
Diffusion policies as an expressive policy class for offline reinforcement learning ICLR 2023 Code
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion RSS 2023 Project
Is conditional generative modeling all you need for decision-making? ICLR 2023 -
Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation CVPR 2024 Project
Dexterous Functional Pre-Grasp Manipulation with Diffusion Policy arXiv 2024 Project
Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals ICRAW 2024 -
Language-Guided Object-Centric Diffusion Policy for Collision-Aware Robotic Manipulation arXiv 2024 -
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies RSS 2023 Project

Policy Learning

Reinforcement Learning

Paper Venue Code/Project
Learning dexterous in-hand manipulation IJRR 2020 -
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot IROS 2023 Code
Rapid Motor Adaptation for Robotic Manipulator Arms CVPR 2024 -
SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering RSS 2023 Project
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation CoRL 2021 Project
Demonstration-Guided reinforcement learning with efficient exploration for task automation of surgical robot ICRA 2023 Code
Text2Reward: Reward Shaping with Language Models for Reinforcement Learning ICLR 2024 Project
Eureka: Human-Level Reward Design via Coding Large Language Models ICLR 2024 Project

Imitation Learning

Paper Venue Code/Project
Behavioral Cloning from Observation IJCAI 2018 -
Maximum Entropy Inverse Reinforcement Learning AAAI 2008 -
SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks ICRA 2024 Project
Transformer-based deep imitation learning for dual-arm robot manipulation IROS 2021 Project
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware RSS 2023 Project
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations CoRL 2023 Project
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets RSS 2022 Project
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers RSS 2024 Project
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations ICRA 2024 Project
Learning Manipulation by Predicting Interaction RSS 2024 Project
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation ICRA 2022 Project
Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation ICRA 2023 Project
Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning ICLR 2023 Code
Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation CVPR 2024 Project
EDGI: Equivariant Diffusion for Planning with Embodied Agents NeurIPS 2023 -
SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion ICRA 2023 Project

Other Methods

Paper Venue Code/Project
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy CVPR 2023 Project
UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning ICCV 2023 -
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning arXiv 2023 Project
Grounding Multimodal Large Language Models in Actions arXiv 2024 -
OpenVLA: An Open-Source Vision-Language-Action Model arXiv 2024 Project
3D-VLA: A 3D Vision-Language-Action Generative World Model ICML 2024 Code
RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models ICML 2024 Project

Embodied Task-Oriented Learning

Object Grasping

Single-Object Grasping

Paper Venue Code/Project
Deep Learning for Detecting Robotic Grasps IJRR 2015 -
Real-Time Grasp Detection Using Convolutional Neural Networks ICRA 2015 -
RGB-D Object Recognition and Grasp Detection using Hierarchical Cascaded Forests TRO 2017 -
FANet: Fast and Accurate Robotic Grasp Detection Based on Keypoints TASE 2023 -
Anygrasp: Robust and efficient grasp perception in spatial and temporal domains TRO 2023 -
Dynamic grasp and trajectory planning for moving objects Autonomous Robots 2019 -
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach RSS 2018 Code
DITTO: Demonstration Imitation by Trajectory Transformation Autonomous Robots 2019 Project
D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions CVPR 2022 Project
GraspXL: Generating Grasping Motions for Diverse Objects at Scale ECCV 2024 Project

Transparent Object Grasping.

Paper Venue Code/Project
Transcg: A large-scale real-world dataset for transparent object depth completion and a grasping baseline RAL 2022 Project
Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects CoRL 2021 Project
Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects CoRL 2022 Project
Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf ICRA 2023 Project
NFL: Normal Field Learning for 6-DoF Grasping of Transparent Objects RAL 2023 Project
TRansPose: Large-scale multispectral dataset for transparent object IJRR 2024 Project
TODE-Trans: Transparent Object Depth Estimation with Transformer ICRA 2023 Code

Grasping in Clutter.

Paper Venue Code/Project
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes ICRA 2021 Project
CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation ICRA 2022 Project
6-DOF Grasping for Target-driven Object Manipulation in Clutter ICRA 2020 -
DDGC: Generative Deep Dexterous Grasping in Clutter RAL 2021 Project
Graspness Discovery in Clutters for Fast and Accurate Grasp Detection ICCV 2021 -
Discriminative Active Learning for Robotic Grasping in Cluttered Scene RAL 2023 -
Efficient learning of goal-oriented push-grasping synergy in clutter RAL 2021 Code
Robust object grasping in clutter via singulation ICRA 2019 -
A deep learning approach to grasping the invisible RAL 2020 Project
A joint modeling of vision-language-action for target-oriented grasping in clutter ICRA 2023 Code

Dynamic Object Grasping.

Paper Venue Code/Project
Predicting Human Intentions in Human–Robot Hand-Over Tasks Through Multimodal Learning TASE 2021 -
Reactive Human-to-Robot Handovers of Arbitrary Objects ICRA 2021 Project
Flexible Handover with Real-Time Robust Dynamic Grasp Trajectory Generation IROS 2023 -
Learning Human-to-Robot Handovers from Point Clouds CVPR 2023 Project
GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation Demonstration and Imitation CVPR 2024 Project
SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers ICRA 2024 Project
Dynamic Grasping with Reachability and Motion Awareness IROS 2021 Project
Target-referenced Reactive Grasping for Dynamic Objects CVPR 2023 Project
ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation 3DV 2024 Project

Multi-Object Grasping

Paper Venue Code/Project
Multi-Object Grasping in the Plane ISRR 2022 Project
Multi-Object Grasping - Types and Taxonomy ICRA 2022 -
Learning to Efficiently Plan Robust Frictional Multi-Object Grasps IROS 2023 -
Multi-Object Grasping—Experience Forest for Robotic Finger Movement Strategies RAL 2024 -
Push-MOG: Efficient Pushing to Consolidate Polygonal Objects for Multi-Object Grasping CASE 2023 Project
Exploiting Kinematic Redundancy for Robotic Grasping of Multiple Objects TRO 2023 -
Grasp multiple objects with one hand RAL 2024 Project

Object Manipulation

Non-Dexterous Manipulation

Paper Venue Code/Project
Pick and place without geometric object models ICRA 2018 -
Self-supervised Learning for Precise Pick-and-place without Object Model RAL 2020 Project
SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphs ICRA 2024 Project
Form2fit: Learning shape priors for generalizable assembly from disassembly ICRA 2020 Project
Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation RSS 2022 Project
Learning where to trust unreliable models in an unstructured world for deformable object manipulation Science Robotics 2021 -
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts CVPR 2023 Project
UniDoorManip: Learning Universal Door Manipulation Policy Over Large-scale and Diverse Door Manipulation Environments arXiv 2024 Project
PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations CVPR 2023 Project
Look Closer: Bridging Egocentric and Third-Person Views With Transformers for Robotic Manipulation RAL 2022 Project
Self-Corrected Multimodal Large Language Model for End-to-End Robot Manipulation arXiv 2024 Project

Dexterous Manipulation

Paper Venue Code/Project
DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation CoRL 2023 Project
MyoDex: A Generalizable Prior for Dexterous Manipulation ICML 2023 Project
Deep Dynamics Models for Learning Dexterous Manipulation CoRL 2020 Project
Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation ICRA 2023 Project
Visual Dexterity: In-hand Dexterous Manipulation from Depth ICMLW 2023 -
Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics ICLR 2023 Project

Tool Manipulation.

Paper Venue Code/Project
Tactile Tool Manipulation ICRA 2023 -
Understanding Tools: Task-Oriented Object Modeling, Learning and Recognition CVPR 2015 -
How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning RAL 2021 -
KETO: Learning Keypoint Representations for Tool Manipulation ICRA 2020 -
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision IJRR 2020 Project
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools CoRL 2023 Project
Leveraging Language for Accelerated Learning of Tool Manipulation ICLR 2022 Project
Creative robot tool use with large language models arXiv 2023 Project

Datasets

Datasets for Object Grasping

Paper Name Venue Code/Project
Efficient grasping from rgbd images: Learning using a new rectangle representation Cornell ICRA 2011 -
Real-World Multiobject, Multigrasp Detection Multi-Object RAL 2018 Code
Jacquard: A Large Scale Dataset for Robotic Grasp Detection Jacquard IROS 2018 Project
Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations VR-Grasping-101 ICRA 2018 Project
ACRONYM: A Large-Scale Grasp Dataset Based on Simulation ACRONYM ICRA 2021 Project
Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation EGAD RAL 2020 Project
Graspnet-1billion: A large-scale benchmark for general object grasping GraspNet-1Billion CVPR 2020 Project
Grasp-Anything: Large-scale Grasp Dataset from Foundation Models Grasp-Anything ICRA 2024 Project

Datasets for Object Manipulation

Paper Name Venue Code/Project
Yale-CMU-Berkeley dataset for robotic manipulation research YCB IJRR 2017 Project
AKB-48: A Real-World Articulated Object Knowledge Base Multi-Object CVPR 2022 Project
SAPIEN: A SimulAted Part-based Interactive ENvironment PartNet-Mobility CVPR 2020 Project
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts GAPartNet CVPR 2023 Project
Maniskill2: A unified benchmark for generalizable manipulation skills ManiSkill2 ICLR 2023 Project
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes ARNOLD ICCV 2023 Project
Bi-DexHands: Towards Human-Level Bimanual Dexterous Manipulation Bi-DexHands TPAMI 2023 Code
DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects DexArt CVPR 2023 Project
Grasp-Anything: Large-scale Grasp Dataset from Foundation Models Grasp-Anything ICRA 2024 Project
PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations PartManip CVPR 2023 Project
BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation BEHAVIOR-1K CoRL 2022 Project

Applications

Industrial Robots

Paper Venue Code/Project
Robot learning of industrial assembly task via human demonstrations Autonomous Robots 2019 -
Applying a 6 DoF robotic arm and digital twin to automate fan-blade reconditioning for aerospace maintenance, repair, and overhaul Sensors 2020 -
Advanced predictive maintenance with machine learning failure estimation in industrial packaging robots DAS 2020 -
Compound fault diagnosis for industrial robots based on dual-transformer networks Manu. Sys. 2023 -

Agricultural Robots

Paper Venue Code/Project
Technological revolutions in smart farming: Current trends, challenges & future directions Comput Electron Agr 2022 -
Robotics in agriculture: Advanced technologies in livestock farming and crop cultivation E3S Web Conf. 2024 -
State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: A review Comput Electron Agr 2020 -
Current status and future challenges in implementing and upscaling vertical farming systems Nature Food 2021 -
Intelligent robots for fruit harvesting: Recent developments and future challenges Precis Agric 2022 -

Domestic Robots

Paper Venue Code/Project
Learning arbitrary-goal fabric folding with one hour of real robot experience CoRL 2021 Project
RCareWorld: A Human-centric Simulation World for Caregiving Robots IROS 2022 Project
Improving Robotic Cooking Using Batch Bayesian Optimization RAL 2022 -
Robot Cooking With Stir-Fry: Bimanual Non-Prehensile Manipulation of Semi-Fluid Objects RAL 2022 -
Efficient Task/Motion Planning for a Dual-arm Robot from Language Instructions and Cooking Images IROS 2022 -
Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization RAL 2024 -

Surgical Robots

Paper Venue Code/Project
Autonomous tissue manipulation via surgical robot using learning based model predictive control ICRA 2019 Project
Super deep: A surgical perception framework for robotic tissue manipulation using deep learning for feature extraction ICRA 2021 -
Transition state clustering: Unsupervised surgical trajectory segmentation for robot learning IJRR 2017 Project
SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning IROS 2021 Code
Towards Human-Robot Collaborative Surgery: Trajectory and Strategy Learning in Bimanual Peg Transfer RAL 2023 -
Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learning RAL 2023 Project
AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy MICCAI 2022 Project
ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity ICRA 2024 Project

Other Applications

Paper Venue Code/Project
Reinforcement learning in dual-arm trajectory planning for a free-floating space robot Aerosp Sci Technol 2020 Project
Robust Adaptive Learning Control of Space Robot for Target Capturing Using Neural Network TNNLS 2023 -
RLBench: The Robot Learning Benchmark & Learning Environment RAL 2020 Project

Challenges and Future Directions

Sim-to-Real Generalization

Paper Venue Code/Project
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping ICRA 2018 Project
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization ICRA 2018 -
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World IROS 2017 Project
On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer RAL 2024 -
TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction arXiv 2024 Project
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation CVPR 2024 Project
Generalizing 6-DoF Grasp Detection via Domain Prior Knowledge RSS 2024 Project
Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking ECCV 2022 Project
Kalman Filter-Based One-Shot Sim-to-Real Transfer Learning RAL 2023 -

Multimodal Embodied LLMs

Paper Venue Code/Project
Embodied Task Planning with Large Language Models arXiv 2023 Project
PaLM-E: An Embodied Multimodal Language Model ICML 2023 Project
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation CVPR 2024 Project
Reasoning Tuning Grasp: Adapting Multi-Modal Large Language Models for Robotic Grasping CoRLW 2023 Project
ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models IROS 2024 Code
A3VLM: Actionable Articulation-Aware Vision Language Mode arxiv 2024 Code

Human-Robot Collaboration

Paper Venue Code/Project
Progress and Prospects of the Human-Robot Collaboration Autonomous Robots 2018 -
A Learning Based Hierarchical Control Framework for Human–Robot Collaboration TASE 2022 -
Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration CoRL 2022 Project

Model Compression and Robot Acceleration

Paper Venue Code/Project
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control CoRL 2023 Project

Model Interpretability and Application Safety

Paper Venue Code/Project
Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation IROS 2019 Project
Efficient and Interpretable Robot Manipulation with Graph Neural Networks RAL 2022 -
Interpretable Robotic Manipulation from Language arXiv 2024 Code
Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark NeurIPS 2023 Project
Physical Adversarial Attack on a Robotic Arm RAL 2022 -
SMOF - A Safety MOnitoring Framework for Autonomous Systems TSMC 2016 -

đź“š License

This repository is released under the MIT license.

About

A Survey of Embodied Learning for Object-Centric Robotic Manipulation

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •