Skip to content

[WACV 2023] Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand

License

Notifications You must be signed in to change notification settings

SHI-Labs/FcF-Inpainting

Repository files navigation

FcF-Inpainting

Open In Colab Huggingface space Framework: PyTorch License

Jitesh Jain, Yuqian Zhou, Ning Yu, Humphrey Shi, WACV 2023

Equal Contribution

[Project Page] [arXiv] [pdf] [BibTeX]

This repo contains the code for our paper Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand.

FcFGAN

News

  • [October 6, 2022]: You can host your own FcF-Inpainting demo using streamlit by following the instructions here.
  • [September 5, 2022]: FcF-Inpainting is now available in the image inpainting tool Lama Cleaner. Thanks to @Sanster for integrating FcF-Inpainting into Lama Cleaner!
  • [August 16, 2022]: FcF-Inpainting is accepted to WACV 2023!
  • [August 5, 2022]: Project Page, ArXiv Preprint and GitHub Repo are public!

Contents

  1. Setup Instructions
  2. Dataset Preparation
  3. Training and Evaluation
  4. Citing FcF-Inpainting

1. Setup Instructions

  • Clone the repo:

    git clone https://github.com/SHI-Labs/FcF-Inpainting.git
    cd FcF-Inpainting
  • Create a conda environment:

    conda create --name fcfgan python=3.7
    conda activate fcfgan
  • Install Pytorch 1.7.1 and other dependencies:

    pip3 install -r requirements.txt
    export TORCH_HOME=$(pwd) && export PYTHONPATH=.
  • Download the models for the high receptive perceptual loss:

    mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
    wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth

2. Dataset Preparation

CelebA-HQ Dataset

Training Data

  • Download data256x256.zip from gdrive.

    mkdir -p datasets/
    # unzip & split into train/test/visualization
    bash tools/prepare_celebahq.sh
    
    datasets
    ├── celeba-hq-dataset
    │   ├── train_256
    │   ├── val_source_256
    │   ├── visual_test_source_256

Evaluation Data

  • Generate 2k (image, mask) pairs to be used for evaluation.

    bash tools/prepare_celebahq_evaluation.sh

Places2 Dataset

Training Data

  • Download the Places2 dataset:

    mkdir -p datasets/
    mkdir datasets/places2_dataset/
    wget http://data.csail.mit.edu/places/places365/train_large_places365challenge.tar
    tar -xvf train_large_places365challenge.tar -C datasets/places2_dataset/
    mv datasets/places2_datasets/data_large datasets/places2_dataset/train
    
    wget http://data.csail.mit.edu/places/places365/val_large.tar
    tar -xvf val_large.tar -C datasets/places2_dataset/
    mv datasets/places2_dataset/val_large datasets/places2_dataset/val
    
    datasets
    ├── places2_dataset
    │   ├── train
    │   ├── val
  • Generate 10k (image, mask) pairs to be used for validation during training.

    bash tools/prepare_places_val.sh

Evaluation Data

Irregular Mask Strategy
  • Generate 30k (image, mask) pairs to be used for evaluation.

    bash tools/prepare_places_evaluation.sh
Segmentation Mask strategy
  • Install Detectron2-v0.5.

    python -m pip install detectron2==0.5 -f \
    https://dl.fbaipublicfiles.com/detectron2/wheels/cu110/torch1.7/index.html
  • Download networks for segmentation masks:

    mkdir -p ade20k/ade20k-resnet50dilated-ppm_deepsup/
    wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/encoder_epoch_20.pth
    wget -P ade20k/ade20k-resnet50dilated-ppm_deepsup/ http://sceneparsing.csail.mit.edu/model/pytorch/ade20k-resnet50dilated-ppm_deepsup/decoder_epoch_20.pth
  • Generate (image, mask) pairs to be used for segmentation mask based evaluation.

    bash tools/prepare_places_segm_evaluation.sh

Note: The pairs are only generated for images with detected instances.

3. Training and Evaluation

places

Training on 256x256

  • Execute the following command to start training for 25M images on 8 gpus with 16 images per gpu:

    python train.py \
        --outdir=training-runs-inp \
        --img_data=datasets/places2_dataset/train \
        --gpus 8 \
        --kimg 25000 \
        --gamma 10 \
        --aug 'noaug' \
        --metrics True \
        --eval_img_data datasets/places2_dataset/evaluation/random_segm_256
        --batch 128

Note: If the process hangs on Setting up PyTorch plugin ..., refer to this issue.

Evaluation

Pretrained Models

checkpoint Description
places_512.pkl Model trained on 512x512 for 25M Places2 images
places.pkl Model trained on 256x256 for 25M Places2 images
celeba-hq.pkl Model trained on 256x256 for 25M CelebA-HQ images
  • Run the following command to calculate the metric scores (fid, ssim and lpips) using 8 gpus:

    python evaluate.py \
        --img_data=datasets/places2_dataset/evaluation/random_segm_256 \
        --network=[path-to-checkpoint] \
        --num_gpus=8

celeba

Demo

  • Run the following command and find the results in the visualizations/ folder:

    python demo.py \
    --img_data=datasets/demo/places2 \
    --network=[path-to-checkpoint] \
    --resolution 256

4. Citing FcF-Inpainting

@inproceedings{jain2022keys,
  title={Keys to Better Image Inpainting: Structure and Texture Go Hand in Hand},
  author={Jitesh Jain and Yuqian Zhou and Ning Yu and Humphrey Shi},
  booktitle={WACV},
  year={2023}
} 

Acknowledgement

Code is heavily based on the following repositories: stylegan2-ada-pytorch and lama.

Releases

No releases published

Packages

No packages published