Demo2Code

This is an official implementation for the NeurIPS 2023 paper:
Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought

Huaxiaoyue (Yuki) Wang · Gonzalo Gonzalez-Pumariega · Yash Sharma . Sanjiban Choudhury

Table of Contents

1. Installation
   • Installation
   • Navigating the Repo
2. Quick Start
3. Detailed Usage
   • Part 1: Generate Code
   • Part 2: Execute Code
   • Part 3: Evaluate
4. Contributing
5. License
6. Contact
7. Acknowledgments

Getting Started

Installation

1. Create the conda env from yaml file

conda create -n demo2code python=3.8.16

2. Clone the robotouille repo somewhere on your computer (Not in this repo!)

https://github.com/portal-cornell/robotouille/tree/main

3. cd into robotouille's folder. check out to git checkout 4190a1deabcd108320e6e4c694ba7fad55ea5b76

4. activate the conda env: conda activate demo2code

5. run pip install -e .

6. cd back to this repo. run pip install -r requirements.txt

Install assets for tabletop environment

Go to folder data/tabletop/env_setup_data and run the following commands:

gdown --id 1Cc_fDSBL6QiDvNT4dpfAEbhbALSVoWcc
gdown --id 1yOMEm-Zp_DL3nItG9RozPeJAmeOldekX
gdown --id 1GsqNLhEl9dd4Mc3BM0dX3MibOI1FVWNM

unzip ur5e.zip
unzip robotiq_2f_85.zip
unzip bowl.zip

(back to top)

Navigating the Repo

• data contains the demonstrations and the prompt files. For each domain, there are folders for:
  • demonstration_config contains
    • _demo-config.txt which contains the list of tasks to run an approach on (to generate code)
    • _test-time_demo-config.txt which contains the list of tasks to run the code generated by the approach on (to evaluate the code)
  • demonstration contains the demonstrations for each approach
  • prompt contains the prompt files for each approach
• outputs contains the output when running the experiments. For each domain, there are folders for:
  • cot contains the output of the recursive summarization (only for demo2code and demonolang2code)
  • code contains the generated code
  • state contains the generated states after running the code (only for robotouille and tabletop)
  • eval_result contains the evaluation results
  • env-info contains the environment information (used for evaluation). This is not saved on the repo, but is generated when running the experiments.
• scripts contains the main python files to run the experiments
  • main demo2code algorithm is in scripts/overall_helpers/code_gen_helper.py

(back to top)

Quick Start

A quick start guide for gonzalo:

You can run everything

python main.py ROBOTOUILLE --model_names demo2code --render

Alternatively, if you just want to render the code

python scripts/robotouille_main.py --model_names demo2code --render

A couple of things you need to make sure:

• In data/robotouille/prompt/demo2code/demo2code_prompt.yaml, you need to verify the prompt_version.
  • This prompt_version affects what folder the results get stored.
  • For example, the generated states will be stored under outputs/robotouille/state/demo2code/{prompt_version}
• To modify what tasks get run/tested, you should modify data/robotouille/demonstration_config/latest-test-time_demo-config.txt.
  • here "latest" correspond to the demo_version.
  • this affects what demo gets loaded/used in data/robotouille/demonstration/demo2code/{demo_version}
  • to change demo_version (not recommended), you can use the argument --demo_version when you run the python file

python scripts/robotouille_main.py --demo_version {put_your_demo_version_here} --model_names demo2code --render

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Yuki Wang - yukiwang@cs.cornell.edu

Paper Project Link: https://portal-cornell.github.io/demo2code-webpage/

(back to top)

Acknowledgments

We thank Nicole Thean (@nicolethean) for her help with creating the assets that bring Robotouille to life!