Robotics Development with a full fledged Assistant Agent capable of GUI navigation

[Umar-Azam]

The integration of LLM into development workflows is super interesting. I have been looking at some recent work with agents that utilize retrieval augmented generation and knowledge files. And some github projects leveraging the GPT4-Vision model that are developing agents capable of GUI navigation like the self operating computer .

I am expecting the capabilities of vision-language models to significantly improve over the next year, enabling improved versions of agents capable of interfacing with the terminal like open interpreter and with the GUI interface like the self-operating computer.

This should in theory enable autonomous robotics development if all the documentation is fed into specialized agent systems as knowledge files. Some early work using LLMs to develop reinforcement learning policies in shown in the EUREKA paper for dexterous manipulation and pen tricks. Newer, more capable multi-modal models should enable more advanced automation of development workflows.

Is anyone working on this or thinking about this ? I am currently trying to research what kind of architecture such an agent would have and how such development workflows would be automated.

[fujitatomoya]

@Umar-Azam thanks for sharing the information, those are really interesting, i will take a closer look into them. i would say ros2ai is more dedicated to application framework for ROS 2 and more system perspective helper, but those information really helps it for sure.