This repository is the official Pytorch implementation for Smooth Diffusion.
Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models
Jiayi Guo*, Xingqian Xu*, Yifan Pu, Zanlin Ni, Chaofei Wang, Manushree Vasu, Shiji Song, Gao Huang, Humphrey Shi
Smooth.diff-nonloop-0319.mp4
Smooth Diffusion is a new category of diffusion models that is simultaneously high-performing and smooth.
Our method formally introduces latent space smoothness to diffusion models like Stable Diffusion. This smoothness dramatically aids in: 1) improving the continuity of transitions in image interpolation, 2) reducing approximation errors in image inversion, and 3) better preserving unedited contents in image editing.
- [2024.09.24] Since SD 1.5 is currently unavailable, we set Realistic_Vision_V2.0 as our default model.
- [2024.03.25] Our demo is available on 🤗 Huggingface Space!
- [2024.03.20] Code, model, and demo released!
- [2024.02.27] Smooth Diffusion is accepted by CVPR 2024!
- [2023.12.08] Paper released!
- ☑️ Release code and model weights
- ☑️ Gradio Demo
Smooth Diffusion (c) enforces the ratio between the variation of the input latent and the variation of the output prediction is a constant. We propose Training-time Smooth Diffusion (d) to optimize a "single-step snapshot" of the variation constraint in (c). DM: Diffusion model. Please refer to our paper for additional details.
conda create --name smooth-diffusion python=3.9
conda activate smooth-diffusion
pip install torch==2.0.0 torchvision==0.15.1 torchaudio==2.0.1
pip install -r requirements.txt
We provide a WebUI empowered by Gradio. Start the WebUI with the following command:
python app.py
We provide scripts for data downloading and training. Unfortunately, the LAION dataset is currently unavailable due to safety review. [Offcial note by LAIOM.ai] (Update: LAION is available now. Have fun!)
# Download LAION aesthetics 6.5+
python download_regularization_images.py
# Train smooth LoRA
bash train.sh
Using the Smooth LoRA trained atop Stable Diffusion V1.5.
Integrating the above Smooth LoRA into other community models.
If you find our work helpful, please star 🌟 this repo and cite 📑 our paper. Thanks for your support!
@InProceedings{guo2024smooth,
title={Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models},
author={Jiayi Guo and Xingqian Xu and Yifan Pu and Zanlin Ni and Chaofei Wang and Manushree Vasu and Shiji Song and Gao Huang and Humphrey Shi},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2024}
}
We thank Diffusers (LoRA finetuning) and AlignSD (data downloading).
guo-jy20 at mails dot tsinghua dot edu dot cn