Skip to content

[ICCV2023] Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation

License

Notifications You must be signed in to change notification settings

jixiang2016/PMBNet

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

17 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation

visitors Paper | Supp | Video

Xiang Ji1, Zhixiang Wang1,2, Zhihang Zhong1,2, Yinqiang Zheng1

1The University of Tokyo  2National Institute of Informatics  

TL;DR

In this paper, we present the first real-world dataset for learning and benchmarking degraded video frame interpolation, named RD-VFI, and further explore the performance differences of three types of degradations, including GS blur, RS distortion, and an in-between effect caused by the rolling shutter with global reset (RSGR), thanks to our novel quad-axis imaging system. Moreover, we propose a unified Progressive Mutual Boosting Network (PMBNet) model to interpolate middle frames at arbitrary time for all shutter modes. Its disentanglement strategy and dual-stream correction enable us to adaptively deal with different degradations for VFI.

image

Dependencies

  1. Python and Pytorch
  • Pyhotn=3.8 (Anaconda recommended)
  • Pytorch=1.11.0
  • CUDA=11.3/11.4
conda create -n pmbnet python=3.8
conda activate pmbnet
conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 cudatoolkit=11.3 -c pytorch

####  please install detron2 using command as bellow
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
  1. Other packages
pip install -r requirements.txt

Data and Pretrained Model

  • Download datasets RD_VFI and synthetic data GOPRO-VFI_copy based on GOPRO.
  • Unzip them under a specified directory by yourself.
  • Please download checkpoints from this link and put them under root directory of this project.

Test

To test PMBNet, please run the command below:

bash ./test.sh       ### Please specify your data directory, data mode and output path in the script

Train

To train PMBNet, please run the command below:

bash ./train.sh       ### Please refer to the script for more info.

Citation

If you find our work useful, please kindly cite as:

@InProceedings{Ji_2023_ICCV,
    author    = {Ji, Xiang and Wang, Zhixiang and Zhong, Zhihang and Zheng, Yinqiang},
    title     = {Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {12259-12268}
}

Acknowledgement

This project is based on works below:

About

[ICCV2023] Rethinking Video Frame Interpolation from Shutter Mode Induced Degradation

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published