Code for ‘Drawing’ the inner world of a story using GauGAN in a real environment.
- Make sure you are using an SSH key to connect to GitHub.
Clone the repository including its submodules
git clone --recurse-submodules git@github.com:Marrow-AI/gan-deeplab-spade.git
You can safely ignore the error Failed to recurse into submodule path 'SPADE'.
Create and activate a conda environment from the suppplied file
cd gan-deeplab-spade
conda env create --file environment.yml
conda activate deeplab-spade
Note: The supplied environment requires a CUDA enabled GPU device. It may be possible to create your own environment without GPU acceleration.
Download the DeepLabV2 CocoStuff checkpoint:
mkdir -p checkpoints/deeplab
gdown --id 18kR928yl9Hz4xxuxnYgg7Hpi36hM8J2d -O checkpoints/deeplab/deeplabv2_resnet101_msc-cocostuff164k-100000.pth
Download and extract the SPADE checkpoint:
mkdir -p checkpoints/spade
gdown --id 12gvlTbMvUcJewQlSEaZdeb2CdOB-b8kQ -O checkpoints/spade/checkpoints.tar.gz
tar xvf checkpoints/spade/checkpoints.tar.gz -C checkpoints/spade/
Run the demo to generate images from your webcam:
./demo_spade.sh
The demo transforms bottles (label ID 43) into flowers (label ID 118) in this line:
labelmap[labelmap == 43] = 118
Mappings between IDs and labels are found here.
- Feel free to use the code for any purpose, but we request that you attribute the work to shirin anlen and Avner Peled of the Marrow project.
- Deeplab pytorch implementation by Kazuto Nakashima.
- GauGAN SPADE implementation by NVIDIA Corporation.
Thank you and enjoy!