Face Preserving Style Transfer

Style	Content	Content Stylized Without Facial Preservation	Content Stylized With Facial Preservation

Fast style transfer with facial preservation.

Abstract

Style transfer is the act of stylizing an input image based on the style of another image [Gatys et al. 2015]. We extend the Fast Style Transfer network created by Johnson et al. to include an extra perceptual loss term for the loss in facial details [2016]. The face loss is calculated by using Multi-task Cascaded Convolutional Networks [Zhang et al. 2016] to find faces and OpenFace [Amos et al. 2016] to compute a 128-dimension face descriptor. The loss is then the squared distance between face descriptors.

Created by Zachary Ferguson for CSCI-GA.2271: Computer Vision (Fall 2018) at New York University.

Report

• Compressed version (30 MB): ferguson-zachary-report-small.pdf
• Full Version (86 MB): ferguson-zachary-report.pdf

Model

We train an image transformation network to stylize images based on the style of a target style image. The image transformation network is trained using the perceptual loss computed by neural network(s). We extend the original network of Johnson et al. (circled in red) by including an additional face loss term computed by a new face loss network (circled in grey).

Usage

We implement our network in Python using PyTorch. Additionally, Pillow and NumPy are used to handle images and other miscellaneous tasks.

Stylizing Images

python -m src.stylize --content-image [path/to/content.ext] --style-model [path/to/model.pth] --output [path/to/output.png]

Pretrained Model

We provide a number of different pretrained style model in the models directory. The style models include:

• models/grande-jatte.pth: Style of A Sunday Afternoon on the Island of La Grande Jatte by Georges Seurat
  • style image: images/styles/A-Sunday-Afternoon-on-the-Island-of-La-Grande-Jatte.jpg
  • example results: images/results/grande-jatte/
• models/manga.pth: Style of artwork from Fullmetal Alchemist by Hiromu Arakawa
  • style image: images/styles/manga.png
  • example results: images/results/manga/
• models/manga-face.pth: Style of artwork from Fullmetal Alchemist by Hiromu Arakawa with facial preservation
  • style image: images/styles/manga.png
  • example results: images/results/manga-face/
• models/mosaic.pth: Style of a mosaic tiling
  • style image: images/styles/mosaic.jpg
  • example results: images/results/mosaic/
• models/mosaic-face.pth: Style of a mosaic tiling with facial preservation
  • style image: images/styles/mosaic.jpg
  • example results: images/results/mosaic-face/
• models/rains-rustle.pth: Style of Rain's Rustle by Leonid Afremov
  • style image: images/styles/Rain's-Rustle-by-Leonid-Afremov.jpg
  • example results: images/results/rains-rustle/
• models/stary-night.pth: Style of The Stary Night by Vincent van Gogh
  • style image: images/styles/Starry-Night-by-Vincent-Van-Gogh.jpg
  • example results: images/results/stary-night/
• models/wave.pth: Style of The Great Wave off Kanagawa by Hokusai
  • style image: images/styles/Great-Wave-off-Kanagawa.jpg
  • example results: images/results/wave/

Training for New Styles

To train a new style model you need to first download a image dataset (the pretrained models were trained using the COCO 2017 Train Images [118K/18GB], no need for the annotations). Then you can train a model using the following command

python -m src.train --content-images [path/to/content/] --style-image [path/to/style.jpg] --output [path/to/output/] [--face]

where python is Python >=3.5, path/to/content/ is the path to the root of the training dataset, path/to/style.jpg is the image of the style to learn, and --face turns on facial preservation.

Results of Fast Style Transfer

We reimplemented the Fast Style Transfer network presented in "Perceptual Losses for Real-Time Style Transfer and Super-Resolution" [Johnson et al. 2016].

Mosaic Style

Manga Style

Rain's Rustle Style

Great Wave off Kanagawa Style

Results of Face Preserving Style Transfer

Our additional face loss allows us to train a image transformation network that preserves facial details. We train a face preserving network for two styles (the manga and mosaic style).

Manga Style with Facial Preservation

	Without Facial Preservation	With Facial Preservation

Mosaic Style with Facial Preservation

	Without Facial Preservation	With Facial Preservation