A collection of ROS projects and resources utilizing or simplifying the application of foundation models in robotics.
ROS frameworks for embodied intelligence applications.
- RAI - Flexible AI agent framework to develop and deploy Embodied AI features for robots.
- ROSA - AI Agent that can interact with ROS1- and ROS2-based robotics systems using natural language queries.
- ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning
- ROS-LLM (Auromix)
- ROSGPT: Next-Generation Human-Robot Interaction with ChatGPT and ROS
- ROSGPT_Vision: Commanding Robots Using Only Language Models' Prompts
- The Conversation is the Command: Interacting with Real-World Autonomous Robots through Natural Language (TCC-IRoNL)
ROS projects that wraps foundation models with a ROS interface.
- llama_ros - ROS 2 wrapper for llama.cpp.
- whisper_ros - ROS 2 wrapper for whisper.cpp. Also provides Voice Activity Detection.
- ros2_whisper - ROS 2 wrapper for whisper.cpp.
- ros2_sam ROS 1 wrapper for Segment Anything.
- ros_sam - ROS 2 wrapper for Segment Anything.
- grounding_sam_ros - ROS 1 wrapper for Grounded Segment Anything.
- isaac_ros_segment_anything - ROS 2 wrapper for Segment Anything.
- isaac_ros_foundationpose - ROS 2 wrapper for FoundationPose.
- isaac_ros_dnn_stereo_depth - ROS 2 wrapper for Efficient Semi-Supervised stereo disparity (ESS).
- ros2_nanollm - ROS 2 wrapper for streaming VLMs with quantization.
- ROS2-NanoOWL - ROS 2 wrapper for OWL-ViT open vocabulary detection with TensorRT.
- visualnav-transformer-ros2 - ROS 2 port of visualnav-transformer stack (GNM, ViNT and NoMaD)
Tools that can aid in the development, operation, and troubleshooting processes of robots.
- ros2ai - Use ROS 2 CLI through natural language.
- ROSA - Interact with ROS1- and ROS2-based robotics systems using natural language queries.
- ROScribe - Create ROS packages using LLMs.
- explainable_ROS - Generate explanations for the actions performed by an autonomous robot.
Demonstrates interesting usage foundation models in robots.
- Tabletop Handybot - Use foundation models to perform basic tabletop tasks with a robotic arm.
- Utilizing LLMs as a Task Planning Agent for Robotics - Use LLMs for task planning for in-home item retrieval by a mobile robot.
- rai-rosbot-xl-demo - Use the RAI framework to control a ROSbot XL mobile robot.
- rai-agriculture-demo - Use the RAI framework to supervise autonomous tractors.
- remembr - Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robots
- Yell At Your Robot - Improving On-the-Fly from Language Corrections
- Physical Intelligence π0 - A Vision-Language-Action Flow Model for General Robot Control
- Initial efforts are ongoing in the community to implement and train pi-zero-pytorch and its precursor Transfusion
- Contact
@dusty-nvif you are interested in participating or contributing data (seepi-zero-pytorch/issues/2])
- OpenNav: Efficient Open Vocabulary 3D Object Detection for Smart Wheelchair Navigation
Awesome lists of foundation models that can be used in robotics.
- Awesome-LLM - A curated list of Large Language Models.
- Awesome VLM Architectures - A curated list of famous vision language models and their architectures.
- Awesome Robotics Foundation Models - A survey of foundation models in robotics.
- Awesome Foundation and Multimodal Models - Curated list of top foundation and multimodal models