git clone https://github.com/xingy038/Dreamer-XL.git --recursiveOur default, provided install method is based on Conda package. Firstly, you need to create an virtual environment and install the submodoules we provide. (slightly difference from original 3DGS)
conda create -n DreamerXL python=3.9.16 cudatoolkit=11.8
conda activate DreamerXL
pip install -r requirements.txt
pip install submodules/diff-gaussian-rasterization/
pip install submodules/simple-knn/We will provide four templates config for training. (all configs can be trained in a single A100).
The pre-trained model will be downloaded automatically. You can also change model_key: in the configs/<config_file>.yaml to link the local Pretrained Diffusion Models ( Stable Diffusion XL 1.0 in default)
If you using vanilla Stable Diffusion XL 0.9 or 1.0, please make sure to use torch.float32 in guidance/sd_step.py line 81. If you would like to use torch.float16, you can try madebyollin/sdxl-vae-fp16-fix, but this may result in NaN loss. Additionally, the generation results of avatars by vanilla Stable Diffusion XL 0.9 or 1.0 are not satisfactory, so we recommend trying Civitai. The model we tested well is ZavyChromaXL. The generated results are highly dependent on the initial results. Once your generated results are unreasonable, please check your initialization results. By the way, the initialization results of A100 are not particularly good.
python train.py --opt <path to config file>bagel.yaml
batman.yaml
dog.yaml
Iron_Man.yaml@misc{miao2024dreamer,
title={Dreamer XL: Towards High-Resolution Text-to-3D Generation via Trajectory Score Matching},
author={Xingyu Miao and Haoran Duan and Varun Ojha and Jun Song and Tejal Shah and Yang Long and Rajiv Ranjan},
year={2024},
eprint={2405.11252},
archivePrefix={arXiv},
primaryClass={cs.CV}
}This work is built on many amazing research works and open-source projects:
Thanks for their excellent work and great contribution to 3D generation area.