Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting

Yiming Huang *, Beilei Cui *, Long Bai *, Ziqi Guo, Mengya Xu, Hongliang Ren

Medical Image Computing and Computer Assisted Intervention (MICCAI) 2024

|| Paper || Arxiv ||

TODO

• EndoNeRF
• StereoMIS
• Pretrained checkpoints

Environments

We build the Python environment using Anaconda:

git clone https://github.com/lastbasket/Endo-4DGS.git
cd Endo-4DGS
git submodule update --init --recursive
conda create -n ED4DGS python=3.7 
conda activate ED4DGS

pip install -r requirements.txt
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1 --index-url https://download.pytorch.org/whl/cu118

Datasets

We used two datasets for training and evaluation.

EndoNeRF

We used EndoNeRF dataset provided by Yuehao Wang.

StereoMIS

We used StereoMIS dataset from Michel Hayoz

The data structure is as follows:

./data
├── endonerf
│    ├── cutting_tissues_twice
│    │    └── ...
│    ├── pulling_soft_tissues
│    │    └── ...
│    └── ...
└── stereomis
    └── ...

Preparing the Pretrained Depths from Depth-Anything

Download depth_anything_vits14.onnx and place in: ./submodules/depth_anything/weights/depth_anything_vits14.onnx

sh prepare_depth.sh

Training

sh train.sh

More configurations can be found in arguments/$DATASET.py

Evaluation

# Render the testing result
sh render.sh
# Evaluation
sh eval.sh

Inferring the pretrained Gaussians

Download pretrained_gaussians and unzip in the project directory:

# Render the testing result e.g. endonerf/pulling
python render.py --model_path pretrained_gaussians/endonerf/pulling --pc --skip_video --skip_train --configs arguments/endonerf.py

Cite

@misc{huang2024endo4dgsendoscopicmonocularscene,
      title={Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting}, 
      author={Yiming Huang and Beilei Cui and Long Bai and Ziqi Guo and Mengya Xu and Mobarakol Islam and Hongliang Ren},
      year={2024},
      eprint={2401.16416},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2401.16416}, 
}

Acknowledgement

Thanks the authors for their works: StereoMIS, diff-gaussian-rasterization-depth, EndoNeRF, 4DGaussians, Depth-Anything-ONNX.