The architecture of the AssemblyHelper includes a perception module, a reasoning module, and an execution module. When performing manufacturing tasks, the workflow of the HRC system is as follow:
Datasets: The HRCA-Code dataset for HRC task reasoning include 26 simple tasks, 24 medium tasks, and 16 difficult tasks. See here for more details. The HRCA-Env dataset for scene updating is a subset of HRCA-Code dataset, which includes (24-7) medium tasks and 16 hard tasks, excluding too simple tasks and tasks without any code.
Results: We test the HRC-prompt based LLMs performance on the HRC datasets. Exitensive experiments are conducted with distinct prompt strategies, shot number, models, human feedback on HRC task reasoning. For updating scene observations after executing HRC code, We also compare two methods, named union and decomposition. Overall, the performance on the HRC task reasoning and scene updating can be seen in here. For the satistical results on each experiment, it can be found in this EXCEL. For more details about each task's response, please see each folder in here for experiments on different perspectives and the file end with *.yml describes the whole tasks (including task input & llm response) on each setting. Each file is named as [models]_[strategy_1]_[strategy_n]_[shot number]_[scene generation method], such as gpt4_cot_3shot_sot_union_scene.
Datasets: The HRCA-Obj dataset for object classification, including 8 objects. The trainset with about 600 images per catergory. The testset with 40 images per catergory, including half of them occluded. The label feature of dataset and PCA tranform matrix can be accessed in here. The raw images of HRCA-Obj are stored in Alipan with code 2ii1.
Results: The results of VFMs for classification can be found in this EXCEL. The evaluation code and image annotation tools placed in here.
Assets: Some quantity results in HRC assembly is stored in here and here. We also provide a notebook for semantic segmentation in task processes. A webui for HRC task reasoning is in here.