Latest-development-of-ISR-VSR

[Updating...] Papers and related resources, mainly state-of-the-art and novel works in ICCV, ECCV and CVPR about image super-resolution and video super-resolution.

Contents

• 1. Metrics dispute
• 2. Latest survey
• 3. Upscale method
• 4. Unsupervised Super-Resolution Method
• 5. Real-Word Image Super-Resolution
• 6. Stereo Image Super-Resolution
• 7. Image-Super-Resolution (ISR)
• 8. Video-Super-Resolution (VSR)
• 9. Library
• 10. Related Research institutions

Metrics dispute

Suggestion in SR: CVPR2018 "The Perception-Distortion Tradeoff"

Latest survey

• Deep Learning for Image Super-resolution: A Survey, arXiv [Submitted on 16 Feb 2019 (v1), last revised 8 Feb 2020 (this version, v2)], accepted by PAMI2020
• A Deep Journey into Super-resolution: A survey, arXiv, [Submitted on 16 Apr 2019 (v1), last revised 23 Mar 2020 (this version, v3)]
• Deep learning methods in real‑time image super‑resolution: a survey, Journal of Real-Time Image Processing2020
• Survey on Single Image based Super-resolution—Implementation Challenges and Solutions, Multimedia Tools and Applications2020

Upscale method

• Dconvolution: "Deconvolutional networks"
• Sub-pixel: "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network"
• Unpooling: "Visualizing and understanding convolutional networks"
• DUpsample: "Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation"
• Carafe: "CARAFE- Content-Aware ReAssembly of FEatures"
• Meta-SR: "Meta-SR-A Magnification-Arbitrary Network for Super-Resolution"
• Scale-arbitrary SR:Learning for Scale-Arbitrary Super-Resolution from Scale-Specific Networks, arXiv2020

Unsupervised Super-Resolution Method

1. "Zero-Shot" Super-Resolution using Deep Internal Learning, CVPR2018
2. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks, CVPRW2018
3. Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy, Medical image analysis 2019
4. Self-Supervised Fine-tuning for Image Enhancement of Super-Resolution Deep Neural Networks, arXiv2019
5. Unsupervised Learning for Real-World Super-Resolution, arXiv2019
6. Unsupervised Single-Image Super-Resolution with Multi-Gram Loss, MDPI2019

Real-Word Image Super-Resolution

• Based on the proposed HR-LR Image Pairs

1. Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data, ICCVW2021
   codes

2. Toward Bridging the Simulated-to-Real Gap: Benchmarking Super-Reslution on Real Data, TPAMI2019

3. Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model, ICCV2019

4. Camera Lens Super-Resolution, CVPR2019

5. Zoom to Learn, Learn to Zoom, CVPR2019

• Based on the simulated degradation method

6. Finding Discriminative Filters for Specific Degradations in Blind Super-Resolution, NeurIPS2021
   codes

7. Blind Super-Resolution with Iterative Kernel Corrections, CVPR2019

8. Deep Plug-and-Play Super-Resolution for Arbitrary Blur Kernels, CVPR2019

9. Blind Super-Resolution Kernel Estimation using an Internal-GAN, NeurIPS2019

10. Kernel Modeling Super-Resolution on Real Low-Resolution Images, ICCV2019

11. Unsupervised Degradation Representation Learning for Blind Super-Resolution, CVPR2021
    pytorch-codes

12. Flow-based Kernel Prior with Application to Blind Super-Resolution, CVPR2021
    pytorch-codes

Stereo Image Super-Resolution

1. Enhancing the Spatial Resolution of Stereo Images using a Parallax Prior, CVPR2018

2. Learning Parallax Attention for Stereo Image Super-Resolution, CVPR2019

3. Stereoscopic Image Super‑Resolution with Stereo Consistent Feature, AAAI2020 oral

4. A Stereo Attention Module for Stereo Image Super-Resolution, SPL2020

Image Super-Resolution

Sorted by year and the format is: abbreviation, paper title, publicaiton, [highlights], related source code.

In 2021

• A Practical Contrastive Learning Framework for Single Image Super-Resolution, arXiv2021, [contrastive, discriminator, data-augment, task-generalizable embedding]

• GLEAN, GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution, CVPR2021 Oral, [encoder-bank-decoder, StyleGAN as generative latent bank]
  pytorch-codes

• ClassSR, ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic, CVPR2021
  pytorch-codes

• LIIF, Learning Continuous Image Representation with Local Implicit Image Function, CVPR2021
  pytorch-codes

• AdderSR, AdderSR: Towards Energy Efficient Image Super-Resolution, CVPR2021

In 2020

• IPT, Pre-Trained Image Processing Transformer, arXiv2020, [low-level transformer]
  waiting

• IGNN, Cross-Scale Internal Graph Neural Network for Image Super-Resolution, NeurIPS2020, [graph related, patch match]
  codes

• SRFlow, SRFlow: Learning the Super-Resolution Space with Normalizing Flow, ECCV2020
  codes-prepare

• PISR, Learning with Privileged Information for Efficient Image Super-Resolution, ECCV2020, [use encoder and decoder in teacher, distillation, estimator module]
  pytorch-codes

• Coarse-to-fine cnn for image super-resolution, IEEE TMM2020
  pytorch-codes

• Journey Towards Tiny Perceptual Super-Resolution, ECCV2020

• Lightweight Image Super-Resolution with Enhanced CNN, arXiv2020, Elsevier
  pytorch-codes

• Unpaired Image Super-Resolution using Pseudo-Supervision, CVPR2020

• Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline, CVPR2020
  tensorflow-codes

• Invertible Image Rescaling, ECCV2020, [another method to get more information in the sacaling phase, invertible NN, flow-based, wavelet transform]
  codes

• IGNN, Cross-Scale Internal Graph Neural Network for Image Super-Resolution, arXiv2020, [first use the graph neural network, graph construction and patch aggreagation module, find the k similar neighbor patch]
  codes

• TTSR, Learning Texture Transformer Network for Image Super-Resolution, CVPR2020, [proposed a transformer-based model to do SR, texture transformer]
  [codes-wait]

• CutBlur, Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy, CVPR2020, [new data augmentation method called CutBlur, it not only can tackle SR but other low-level tack like denoising and artifact ramoval, cut-and-paste based on patch, let model to know where to SR and how to SR]
  pytorch-codes

• SPSR, Structure-Preserving Super Resolution with Gradient Guidance, CVPR2020, [Gradient guidance to perserve the information, gradient loss, address the geometric distort]
  pytorch-codes

• UDVD, Unified Dynamic Convolutional Network for Super-Resolution with Variational Degradations, CVPR2020, [try to use one model to address several degreadation, Feature Extraction Network(FRN), Refinement Network(RN), Dynamic Block(DB), dynamic conv by a dynamic kernels some like sub-pixel operation]
  ---

In 2019

• SRFBN, Feedback Network for Image Super-Resolution, CVPR2019, [feedback and a lot of comparation]
  pytorch-codes

• zoom-learn-zoom, Zoom to Learn, Learn to Zoom, CVPR2019, [SR-RAW dataset and CoBi loss, real-word, new direction for SR-RAW datasets and new CoBi loss function for alignment]
  tensorflow-codes

• Camera, Camera Lens Super-Resolution, CVPR2019, [real-word, Create City100 Dataset for real-word application]
  tensorflow-codes

• RealSR, Toward Real-World Single Image Super-Resolution: A New Benchmark and A New Model, ICCV2019, [RealSR dataset, real-word, LP-KPN, New RealSR datasets more flexible and convenient to use]
  caffe-codes

• Simulated-to-Real Gap, Toward Bridging the Simulated-to-Real Gap: Benchmarking Super-Reslution on Real Data, TPAMI2019, [hardware binning, real-word, maybe the method older for it's journal]
  ---

• RankSRGAN, RankSRGAN: Generative Adversarial Networks with Ranker for Image Super- Resolution, CVPR2019, [focus on perceptual quality, and new method to use perceptual metrics named Ranker]
  pytorch-codes

• IMDN, Lightweight Image Super-Resolution with Information Multi-distillation Network, ACM MM2019
  pytorch-codes

In 2018

• WDSR, Wide Activation for Efficient and Accurate Image Super-Resolution, arXiv2018, [widen feature map and WN, weight normalization]
  pytorch-codes

• SRMD, Learning a Single Convolutional Super-Resolution Network for Multiple Degradations, CVPR2018, Degraded Fuzzy Kernel and Noise Level
  matlab-codes

• RDN, Residual Dense Network for Image Super-Resolution, CVPR2018 Spotlight, [local and global Residual, bicubic downsampling, gaussian kernel feature fusing]
  official-codes

• DBPN, Deep Back-Projection Networks For Super-Resolution, CVPR2018, [repeat down and up sample a back mechanism, Back-Projection]
  pytorch-codes

• ZSSR, "Zero-Shot" Super-Resolution using Deep Internal Learning, CVPR2018, [re-sample train test, internally train, zero-shot]
  pytorch-codes

• SFTGAN, Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform, CVPR2018, [semantic probability, semantic SFT]
  pytorch-codes

• EUSR, Deep Residual Network with Enhanced Upscaling Module for Super-Resolution, CVPR2018, [enhanced upscaling module (EUM), change EDSR to EUSR by adding EUM]
  ---

• CARN, Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network, ECCV2018, [fast, cascading block]
  pytorch-codes

• GAN_degradation, To learn image super-resolution, use a GAN to learn how to do image degradation first, ECCV2018, [mainly face test, use GAN to prodecu LR near to nature,]
  ---

• RCAN, Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV2018, [Deep, Residual, Channel Attention, very deep residual block with channel attention using several skip connection and channel weight]
  pytorch-codes

• EPSR, Analyzing Perception-Distortion Tradeoff using Enhanced Perceptual Super-resolution Network, ECCV2018, [has a new metrics idea]
  ---

In 2017

• DRRN, Image Super-Resolution via Deep Recursive Residual Network, CVPR2017, [residual network, combine ResNet and recursive]
  caffe-codes

• LapSRN, Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution, CVPR2017, [Pyramid network new loss to constrain]
  matconvnet-codes | pytorch | tensorflow

• SRDenseNet, Image Super-Resolution Using Dense Skip Connections, ICCV2017, [add dense block to model]
  pytorch-codes

• SRGAN, Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, CVPR2017, [first proposed GAN]
  tensorflow | tensorflow | torch | caffe | tensorflow | keras | pytorch

• EDSR, Enhanced Deep Residual Networks for Single Image Super-Resolution, CVPR2017, [remove BN]
  torch | tensorflow | pytorch

In 2016

• FSRCNN, Accelerating the Super-Resolution Convolutional Neural Network, ECCV2016, [deconvolution fine-tuninig last deconv, Develop SRCNN, add deconv, input image don't need to upsample by bicubic and fine-tune accelerate]
  official-matlab-caffe-codes

• ESPCN, Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, CVPR2016, [sub-pixel Tanh instead Relu Real time, A new way to upsamping: sub-pixel]
  tensorflow-codes | pytorch-codes | caffe-codes

• VDSR, Accurate Image Super-Resolution Using Very Deep Convolutional Networks, CVPR2016, [residual network, deep, Add residual, padding 0 every layer, scale mixture training]
  project | caffe | tensorflow | pytorch

• DRCN, Deeply-Recursive Convolutional Network for Image Super-Resolution, CVPR2016, [Recursive Neural Network, Learn RNN to add recursive and skip input image is interpolation image]
  project | tensorflow

• RED, Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections, NIPS2016, [Encoder-decoder and skip]
  ---

In 2014

• SRCNN, Image Super-Resolution Using Deep Convlutional Network, ECCV2014, [Loss:MSE CNN, has two version 2014 and ex-2016. Milestone in deep learning about SR.Simple three CNN network: patch extraction and representation, non-linear mapping and reconstraction]
  keras-codes

Video Super-Resolution

Sorted by year and the format is: abbreviation, paper title, publicaiton, [highlights], related source code.

In 2022

• DAP, Fast Online Video Super-Resolution with Deformable Attention Pyramid, arXiv2022, [deformable attention pyramid, fast]

• Self-Blind-VSR，Self-Supervised Deep Blind Video Super-Resolution, arXiv2022, [for real bind vsr without HR, auxiliary paired data]
  project

In 2021

• RealBasicVSR, Investigating Tradeoffs in Real-World Video Super-Resolution, arXiv2021, [cleaning module, new videoLQ dataset, real VSR]
  Pytorch-codes

• VSR-transformer, Video Super-Resolution Transformer, arXiv2021, [transformer in VSR]
  Pytorch-codes

• GLEAN, Glean: Generative latent bank for large-factor image super-resolution, CVPR2021 oral, [Generative LatEnt bANk(GLEAN), encoder-bank-decoder architecture with multiresolution skip connections]
  Pytorch-codes

• BasicVSR, BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond, CVPR2021, [BasicVSR architecture, research on essential component]
  Pytorch-codes

In 2020

• STVSR, Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution, CVPR2020, [two task but with one stage, video frame interpolation, bidirectional deformable convLSTM]
  Pytorch-codes

In 2019

• RBPN, Recurrent Back-Projection Network for Video Super-Resolution, CVPR2019, [recurrent encoder-decoder module]
  Pytorch-codes

• EDVR, EDVR: Video Restoration with Enhanced Deformable Convolutional Networks, CVPR2019, [PCD:Pyramid, Cascading and Deformable (PCD) alignment module, TSA:Temporal and Spatial Attention fusion module, proposed two specify modules: PCD and TSA. PCD is for alignment and STA is for fusion. With deformable convolution, self-ensemble and two-stage redfine, it wins all four tracks in the NTIRE19 Challenges for Video]
  Pytorch-codes

In 2018

• FRVSR, Frame-Recurrent Video Super-Resolution, CVPR2018, [use a recurrent approach that passes the previously estimated HR frame as an input for the following iteration. Model includes Fnet and SRNet, Flow estimation, Upscaling flow, Warping previous output, Mapping to LR space, Super-Resolution Warp]
  official-codes

• DUF, Deep Video Super-Resolution Network Using Dynamic Upsampling Filters Without Explicit Motion Compensation, CVPR2018, [Dynamic upsampling filter, Residual Learning, propose a novel end-to-end deep neural network that generates dynamic upsampling filters and a residual image, which are computed depending on the local spatio-temporal neighborhood of each pixel to avoid explicit motion compensation. The model includes filter generation network and residual generation network]
  tensorflow-codes | tensorflow

In 2017

• VESPCN, Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation, CVPR2017, [sub-pixel for video compensation transformer, compensation transformer: compare early fusion, slow fusion and 3D conv]
  pytorch | tensorflow

• SPMC, Detail-revealing Deep Video Super-resolution, ICCV2017, [SPMC: Subpixel Motion Compensation layer, show that proper frame alignment and motion compensation is crucial for achieving high quality results, It includes motion estimate, SPMC layer and Detail Fusion Net]
  tensorflow-codes

In 2015

• BRCN, Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution, NIPS2015, [Two sub-network and three kind conv, use recurrent, It has three conv. Feedforward conv, recurrent conv and conditioned conv. And two sub-network: forward and backward sub-network]
  matlab-codes

Library

• A collection of state-of-the-art video or single-image super-resolution architectures, reimplemented in tensorflow. which has most great papers/models about ISR and VSR. Include some useful tools: some models with pre-trained weights, link of datasets, VSR package which offers a training and data processing framework based on TF or pytorch.

• MMEditing, MMEditing is an open source image and video editing toolbox based on PyTorch. It is a part of the OpenMMLab project

Related Research institutions

• X-Pixel Group, CUHK, NTU, SIAT, SenseTime Our vision is to make the world look clearer