Integrate LLM Agents

[chalo2000]

Overview

This PR adds LLM agent infrastructure into Robotouille including

• language-based observations through Gym
• LLM agents with a prompt_builder for version controlling prompts
  • OpenAI, Google, Anthropic model support
• human agent for collecting demonstrations to convert into in-context examples

Changes Made

Environment

• robotouille.json augmented to include language descriptors to describe predicates and actions in language
• Environment JSONs goal_description field changed from solely readability purposes to also being included as the language goal of the environment
• New environments from paper added including synchronous, asynchronous, and multi-agent environments. Find these under environments/env_generator/examples/

Agent

• Agents added including
  • I/O
  • I/O CoT
  • ReAct
  • BFS (see Speed up step function to allow for classical planners #38)
  • Human (for testing environments and collecting demonstrations)
• Prompt builder for version controlling prompts
  • Prompts used in our paper included under agents/prompt_builder/prompts/
• In-context examples flow for inserting N random few-shot examples into existing prompts
  • Examples used in our paper included under agents/in_context_examples/data/
• gpt-cost-estimator integrated for calculating OpenAI model cost

Misc / Bug Fixes

• Hydra library integrated to simplify configuration management
• Update main.py to incorporate Hydra.
• New logging including videos and text logging (for LLM agent reasoning and human demonstrations in the form of ReAct in-context examples)
• simulator deprecated - use run_robotouille instead which uses the new main.py flow using Hydra
• Added procedural generation for containers
  • Fixed procedural generation bugs
• Custom deepcopy added to RobotouilleCanvas to avoid deepcopying PyGame surfaces (for BFS agent)
• Conditional effects have their effects negated if not met (e.g. stops cooking if an item placed on a stove is picked up)
• Documentation updated and added for new directories

Test Coverage

Tested a fresh environment on Windows and Mac following top-level README.md and agent/ directory README.md

Related PRs or Issues

#37
#38