This is a PyTorch implementation of our work:
Aleksei Staroverov, Kirill Muravyev, Tatiana Zemskova, Dmitry Yudin and Aleksandr I. Panov
AIRI, MIPT, FRCCSC
SkillFusion is the winner of the CVPR 2023 Habitat ObjectNav Challenge.
In recent years, Embodied AI has become one of the main topics in robotics. For the agent to operate in human-centric environments, it needs the ability to explore previously unseen areas and to navigate to objects that humans want the agent to interact with. This task, which can be formulated as ObjectGoal Navigation (ObjectNav), is the main focus of this work. To solve this challenging problem, we suggest a hybrid framework consisting of both not-learnable and learnable modules and a switcher between them -- SkillFusion. The former are more accurate while the latter are more robust to sensors' noise. To mitigate the sim-to-real gap which often arises with learnable methods we suggest training them in a way that they are less environment-dependent. As a result, our method showed both the top results in the Habitat simulator and during the evaluations on a real robot.
Additionaly, we provide a PyTorch implementation of SkillTron -- a method that further refines visual object navigation performance of SkillFusion approach by using a two-level interactive semantic map representation.
For convenience, we provide different containers that can be used to evaluate SkillFusion and SkillTron:
- A docker container for interactive work with our SkillFusion and SkillTron agents via JupyterLab
- Eval-AI submission-like docker containers that will launch a script for evaluation without access to docker container system.
- A Singularity container for evaluation on platforms that do not have Docker.
All necessary files to build and launch an interactive docker container with Jupyter Lab can be found in docker_interactive
To build the correspoding docker image run:
bash build.shThis will create "alstar_docker" docker image.
To start a docker-container from this image run:
bash run.sh <PORT>This will launch a Jupyter Lab server at localhost:<PORT>
You may need to change paths to mounted volumes before starting the container:
-v <path_to_Habitat_sim_data>/data:/data \
-v <path_to_SkillFusion>/root/:/root alstar_dockerWe provide minimal docker images to run evaluation of SkillFusion and SkillTron methods in Eval-AI submission-like mode.
Note: you need to download and put to expected folders pretrained weights before building evaluation containers.
-
SkillFusion
To build the minimal SkillFusion docker image, go to docker_skillfusion:
bash build.sh
This will create "skillfusion_docker" docker image. Here, the code will not be mounted to docker container, but rather copied to the docker image during its built. The image will contain only the code corresponding to SkillFusion agent in the folder /root/exploration_ros_free/.
To evaluate SkillFusion agent run:
bash test_local.sh
You may need to change paths to mounted volumes before starting the container:
-v $(pwd)/habitat-challenge-data:/data \ -
SkillTron
To build the minimal SkillTron docker image, go to docker_skilltron:
bash build.sh
This will create "skilltron_docker" docker image. Here, the code will not be mounted to docker container, but rather copied to the docker image during its built. The image will contain only the code corresponding to SkillTron agent in the folder /root/exploration_ros_free/.
To evaluate SkillTron agent run:
bash test_local.sh
You may need to change paths to mounted volumes before starting the container:
-v $(pwd)/habitat-challenge-data:/data \
We provide a .def file to build a singularity container with necessary dependencies to run SkillFusion and SkillTron agents evaluation.
Go to the folder singularty.
To build the singularity container:
sudo singularity build skill_fusion.sif skill_fusion.def Additionally, we provide the skilltron_eval.sh script that can be used to launch SkillTron evaluation on slurm based clusters. To use it you may need to change paths to home and data directories as well as change partition name.
Follow official instruction to download Habitat-Matterport 3D Research Dataset (HM3D). For evaluation you will need only it's validation part.
- PONI Exploration
- RL Exploration
- RL Goal Reacher
- OneFormer Semantic Predictor for SkillFusion
- SegmATRon Single Frame Baseline Semantic Predictor for SkillTron
- SegmATRon 1 step Semantic Predictor for SkillTron
- SegmATRon 2 steps Semantic Predictor for SkillTron
- SegmATRon 4 steps Semantic Predictor for SkillTron
Expected Data Structure:
skill-fusion
└── root
├── PONI
│ └── pretrained_models
│ └── gibson_models
│ └── poni_seed_234.ckpt
├── skillfusion
│ └── weights
│ ├── config_oneformer_ade20k.yaml
│ ├── model_oneformer_ade20k.pth
│ ├── ex_tilt_4.pth
│ └── grTILT_june1.pth
└── skilltron
└── weights
├── config_oneformer_ade20k.yaml
├── config_single_frame.yaml
├── segmatron_1_step.yaml
├── segmatron_2_step.yaml
├── segmatron_4_steps.yaml
├── single_frame_baseline.pt
├── segmatron_1_step.pt
├── segmatron_2_step.pt
├── segmatron_4_steps.pt
├── ex_tilt_4.pth
└── grTILT_june1.pthTo change versions of SegmATRon go to config_skilltron.yaml and change paths to config file and weights in SemanticPredictor initialization. Available options:
- Single Frame Baseline weights/config_single_frame.yaml, weights
- SegmATRon 1 step weights/segmatron_1_step.yaml, weights
- SegmATRon 2 steps weights/segmatron_2_step.yaml, weights
- SegmATRon 4 steps weights/segmatron_4_steps.yaml, weights
To control real-time or delayed semantic predictions change parameters of semantic predictor in configuration fileconfig_skilltron.yaml:
semantic_predictor:
config_path: 'weights/segmatron_1_step.yaml'
weights: 'weights/segmatron_1_steps_15.pt'
delayed: FalseIf you are inside an interactive shell of the docker/singularity containers run:
cd /root/skillfusion/;
export PYTHONPATH=/root/PONI/:$PYTHONPATH;
python main.py cd /root/skilltron/;
export PYTHONPATH=/root/PONI/:$PYTHONPATH;
python main.pyIf you have built submission-like docker images, you may use provided scripts test_local.sh (see Evalutation Docker Containers (Eval-AI submission-like))