SuRF

SuRF is the first unsupervised method to achieve end-to-end sparsification, to the best of our knowledge. It can utilize the higher-resolution volume to reconstruct higher-frequency geometric details with less computation and memory consumption. Details are described in our paper:

Surface-Centric Modeling for High-Fidelity Generalizable Neural Surface Reconstruction

Rui Peng, Shihe Shen, Kaiqiang Xiong, Huachen Gao, Jianbo Jiao, Xiaodong Gu, Ronggang Wang

ECCV 2024 (arxiv)

📍 If there are any bugs in our code, please feel free to raise your issues.

⚙ Setup

1. Recommended environment

conda create -n surf python=3.10.9
conda activate surf
pip install -r requirements.txt

2. DTU Dataset

We only train our model on DTU dataset. We adopt the full resolution ground-truth depth maps (just for testing) and RGB images, and use the camera parameters prepocessed by CasMVSNet or MVSNet. Simply, please follow the instruction here of UniMVSNet to prepare the dataset. We generate pseudo points and pseudo depths through RC-MVSNet to assist the model optimization, download and unzip them. The final data structure is just like this:

dtu_training                          
  ├── Cameras
    ├── 00000000_cam.txt
    ├── ...             
    ├── pair.txt
  ├── Depths_raw
  ├── Pseudo_depths
  ├── Pseudo_points
  └── Rectified_raw

Rectified_raw is the full resolution RGB images provided in DTU. We use the same training and testing split as SparseNeuS, please refer to here for more details.

For testing, you can download the testing data prepared by SparseNeuS here, which contains some object masks for cleaning the mesh. Put it to <your DTU_TEST path>. For quantitative evaluation, you need to download the ground-truth points from the DTU website and put it to <your GT_POINTS path>.

3. BlendedMVS Dataset

Download BlendedMVS for evaluation. The data structure is just like this:

blendedmvs                          
  ├── 5a0271884e62597cdee0d0eb
    ├── blended_images
    ├── cams
      ├── 00000000_cam.txt
      ├── ...
      └── pair.txt  
  ├── 5a3ca9cb270f0e3f14d0eddb
  ├── ...

4. Tanks and Temples Dataset

Download Tanks and Temples for evaluation. The data structure is just like this:

tanksandtemples                          
  ├── advanced
    ├── Auditorium
    └── ...
  ├── intermediate
    ├── Family
    └── ...

5. ETH3D

Download ETH3D for evaluation following their official instructions. The data structure is just like this:

ETH3D                          
  ├── courtyard
    ├── cams
    └── images
  ├── ...

📊 Testing

1. Download models or precomputed meshes

Download our pretrained model and put it to <your CKPT path>.

CKPT	Meshes	Train Res	Train View	Test Res	Test View	Mean Cham. Dist.↓
surf	meshes	480X640	5 (4src)	576X800	3 (2src)	1.05

You can also download our precomputed DTU meshes through direct inference.

2. DTU testing

Mesh Extraction. We define all information like the model structure and testing parameters in the configuration file. We use the ./confs/surf.conf file for training and testing. You need to first specify the correct values in your own environment, such as <your dataset path> and <your output save path>. You can use our default testing configurations and the model structure. Once everything is ready, you can simply start testing via:

bash ./scripts/run.sh --mode val --resume <your CKPT path>

This will predict all scenes in the test split at view index 23 by default. If you want to get the results at other views (e.g., view43), you can change the ref_view under the val_dataset namespace in configuration file. Meanwhile, you can also specify scene list under the val_dataset namespace to test on a single scene like scene=[scan24,scan55].

Optionaly, you can add --clean_mesh command to generate the filtered mesh, but you need to note that the mask used in --clean_mesh command is from MVSNet and is not the correct object mask used during the quantitative evaluation.

Mesh cleaning. Before evaluation, to generate the clean meshes using the correct mask, you need clean the mesh first:

python evaluation/clean_meshes.py --root_dir <your DTU_TEST path> --out_dir <your output save path>/meshes

Quantitative Evaluation. Run:

python evaluation/dtu_eval.py --dataset_dir <your GT_POINTS path> --out_dir <your output save path>

You need to pay attention to the filename of meshes in evaluation/clean_meshes.py file.

3. BlendedMVS testing

Using the configuration file confs/gens_bmvs.conf to evaluate on BlendedMVS dataset, run:

python main.py --conf confs/gens_bmvs.conf --mode val --resume <your CKPT path> --clean_mesh

Here, we recommand to add the --clean_mesh command. You can change or add more testing scenes through change scene. Note that camera poses in BlendedMVS have a great difference, you need to make sure that the bounding box fits as closely as possible to the object you want to reconstruct, e.g., adjusting factor and num_interval.

4. Tanks and Temples testing

Using the configuration file confs/gens_tanks.conf to evaluate on Tanks and Temples dataset, run:

python main.py --conf confs/gens_tanks.conf --mode val --resume <your CKPT path> --clean_mesh

5. ETH3D testing

Using the configuration file confs/gens_eth3d.conf to evaluate on ETH3D dataset, run:

python main.py --conf confs/gens_eth3d.conf --mode val --resume <your CKPT path> --clean_mesh

⏳ Training & Fine-tuning

1. DTU training

During training, we set the lowest resolution of volume to 64, you need to first specify the value in confs/surf.conf file and run:

bash ./scripts/run.sh --mode train

By default, we employ the DistributedDataParallel mode to train our model on 2 GPUs.

2. DTU fine-tuning

We use confs/surf_finetune.conf file to config the fine-tuning on DTU dataset. For convenience, we use scripts/finetune.sh file to fine-tune all testing scenes at both 23 and 43 views:

bash ./scripts/finetune.sh --resume <your CKPT path>

You can change the scene and view through the --scene and --ref_view command directly or through modifying the configuration file.

You can fine-tune or train on other datasets in a similar manner.

⚖ Citation

If you find our work useful in your research please consider citing our paper:

@inproceedings{peng2024eccv,
  title={Surface-Centric Modeling for High-Fidelity Generalizable Neural Surface Reconstruction},
  author={Peng, Rui and Shihe, Shen and Xiong, Kaiqiang and Gao, Huachen and Jiao, Jianbo and Gu, Xiaodong and Wang, Ronggang},
  booktitle={The 18th European Conference on Computer Vision (ECCV)},
  year={2024}
}

👩‍ Acknowledgements

This code is highly inhreated from GenS, and we use the SDF-based volume rendering in NeuS and blending in IBRNet. Thanks for their great contributions!