3D-Guided Face Manipulation of 2D Images for the Prediction of Post-Operative Outcome after Cranio-Maxillofacial Surgery
This repository yields the code to the corresponding paper *3D-Guided Face Manipulation of 2D Images for the Prediction of Post-Operative Outcome after Cranio-Maxillofacial Surgery (Pubilished at IEEE Transactions on Image Processing).
To install and run our scripts you need:
- Matlab (If you don't have a licence, there might be a workaround (see below))
- A python environment like anaconda
- A Nvidia GPU with CUDA/PyTorch (If you want to train the model, you might need up to 11GB VRAM; 8GB might work too if you lower the batch-size)
- Time to install and download all the submodules, data and config files that this repo requires :)
Choose a directory to install this repo. Then:
git clone --recurse-submodules https://github.com/KIT-IBT/3D-Guided-Face-Mani.git
cd 3D-Guided-Face-Mani
Set up python path:
export PYTHONPATH=$PYTHONPATH$:$(pwd)/3D-Guided-Face-Mani
Install packages. This repo was tested using the following versions. However, more recent versions might just work fine, too.
conda create -n face-mani python=3.7
conda activate face-mani
pip install numpy==1.17.0 \
scipy==1.3.0 \
tqdm==4.34.0 \
opencv-python==4.1.0.25 \
cython==0.29.12 \
scikit-image==0.15.0
Install PyTorch via conda as described here. Tested version: PyTorch==1.6.0, Cuda==10.2, torchvision=0.7.0 Example:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
Go to the 3DDFA website and download the 3DDFA code and extract it. Then, copy the folder to "submodules" e.g.
cp -R ~/Downloads/3DDFA_Release submodules/
cd submodules/3DDFA_Release
- Downloading the Basel Face Model (BFM) on "http://faces.cs.unibas.ch/bfm/main.php?nav=1-0&id=basel_face_model"
- Copy the "01_MorphableModel.mat" file of the BFM to Matlab/ModelGeneration/
- Run the Matlab/ModelGeneration/ModelGenerate.m to generate the shape model "Model_Shape.mat" Note: If you don't have a Matlab licence, you might consider re-writing the ModelGenerate.m file in python (script has only a few lines)!
Note: 3DDFA and Face3D share identical code for rendering and 3DMM processing
cd ../PyTorch3DDFA/utils/cython
python setup.py build_ext -i
cd ../../../face3d/face3d/mesh/cython/
python setup.py build_ext -i
This is a bit hacky: Create uncommited init.py's to let python see the submodules as packages
cd ../../../../..
touch submodules/PyTorch3DDFA/__init__.py submodules/face3d/__init__.py
The model and parameter configs files are storaged in /data/configs
cd ../../../../..
cp submodules/3DDFA_Release/Matlab/ModelGeneration/Model_Shape.mat \
submodules/3DDFA_Release/Matlab/Model_Expression.mat \
submodules/3DDFA/visualize/tri.mat \
data/configs/
Download models from [here](LINK TO IEEE) and copy them to "data/configs"
Go to the 3DDFA website and download the AFLW2000-3D dataset. Then extract it into "data/datasets/AFLW2000"
Go to the 3DDFA website and download the 300W-LP dataset. Then extract it into "data/datasets/300W_LP"
Next, create a small validation dataset to track the training progress:
mkdir data/datasets/300W_LP/validset data/datasets/300W_LP/validset_Flip
mv data/datasets/300W_LP/LFPW/LFPW_image_test_000[0-4]* data/datasets/300W_LP/validset/
mv data/datasets/300W_LP/LFPW_Flip/LFPW_image_test_000[0-4]* data/datasets/300W_LP/validset_Flip/
Create dataset with PNCC renderings and shape modifications. This should run without errors if you set up all configs and models correctly! This might take up to several days. You can speed up this process by creating multiple runs on different subdirectories of the 300W-LP (change "write_dataset.py" accordingly.)
python write_dataset.py 300W-LP
- To use random background images, download the IndoorCVPR dataset from here and extract the folder into "data/datasets/Images"
- Follow the instructions in "submodules/face3d/examples/Data/BFM/readme.md" (Note: If you already downloaded the BFM installed the 3DDFA_Release (see above), then you can skip a few steps. The important thing is that you create the file "BFM.mat" that we need to render random textures from the BFM model)
- Copy the BFM file to the configs
cp submodules/face3d/examples/Data/BFM/Out/BFM.mat data/configs/
- ToDo: Let user create stats_list
- Render synthetic dataset (50k with chin modifications, 50k with nose modifications, 10k+10k with white background). This might take a few hours:
python render_synthetic_dataset.py
To test the model on the AFLW2000 dataset, you have multiple options
E.g. 'nose_1' or 'chin_1' and different scalar multipliers ('p', 'n', or a floating number to linearly scale the size of the chin e.g. '-200000'):
python run_inference.py data/datasets/AFLW2000/image00006.jpg -mult n -mod chin_1 -o data/output_1
python run_inference.py data/datasets/AFLW2000/image00006.jpg -mult p -mod chin_1 -o data/output_1
python run_inference.py data/datasets/AFLW2000/image00006.jpg -mult -200000 -mod chin_1 -o data/output_1
python run_inference.py data/datasets/AFLW2000/image00006.jpg -mult p -mod nose_1 -o data/output_1
The results can be seen in data/output_1
Note: The network works best for frontal faces combined with chin modifications.
However, the results for nose modifications and large pose rotations are often
not very realistic or accurate. Feel free to try to come up with a better model by improving the
training strategy or the network architecture
This will create a prediction grid for every image of the 2000 images in the AFLW2000 dataset
python run_inference.py data/datasets/AFLW2000 -o data/output_2
If you use this software, please cite it as follows:
R. Andlauer and A. Wachter et al., "3D-Guided Face Manipulation of 2D Images for the Prediction of Post-Operative Outcome After Cranio-Maxillofacial Surgery," in IEEE Transactions on Image Processing, vol. 30, pp. 7349-7363, 2021, doi: 10.1109/TIP.2021.3096081.