Skip to content
/ Gen3D Public

[CSUR 2023] A Survey on Deep Generative 3D-aware Image Synthesis

weihaox.github.io/3d-aware-gen/

License

MIT license
156 stars 11 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

weihaox/Gen3D

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

23 Commits
CONTRIBUTING.md
CONTRIBUTING.md
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
index.html
index.html
 
 
jemdoc.css
jemdoc.css
 
 
reference.bib
reference.bib
 
 
save_cited_bib.py
save_cited_bib.py
 
 

Repository files navigation

A Survey on Deep Generative 3D-aware Image Synthesis

ACM Computing Surveys, 2023
Weihao Xia · Jing-Hao Xue

arxiv Paper Project Page CSUR Paper

Introduction

This project lists representative papers/codes/datasets about deep 3D-aware image synthesis. Besides 3D-aware Generative Models (GANs and Diffusion Models) discussed in this survey, this project additionally covers novel view synthesis studies, especially those based on implicit neural representations such as NeRF.

We aim to constantly update the latest relevant papers and help the community track this topic. Please feel free to join us and contribute to the project. Please do not hesitate to reach out if you have any questions or suggestions.

Survey paper

3D Control of 2D GANs

3D Control Latent Directions

For 3D control over diffusion models simiar to GAN, please refer to semantic manipulation in diffusion latent spaces.

  • SeFa: Closed-Form Factorization of Latent Semantics in GANs.
    Yujun Shen, Bolei Zhou.
    CVPR 2021. [Paper] [Project] [Code]

  • GANSpace: Discovering Interpretable GAN Controls.
    Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris.
    NeurIPS 2020. [Paper] [Code]

  • Interpreting the Latent Space of GANs for Semantic Face Editing.
    Yujun Shen, Jinjin Gu, Xiaoou Tang, Bolei Zhou.
    CVPR 2020. [Paper] [Project] [Code]

  • Unsupervised Discovery of Interpretable Directions in the GAN Latent Space.
    Andrey Voynov, Artem Babenko.
    ICML 2020. [Paper] [Code]

  • On the "steerability" of generative adversarial networks.
    Ali Jahanian, Lucy Chai, Phillip Isola.
    ICLR 2020. [Paper] [Project] [Code]

3D Parameters as Controls

  • 3D-FM GAN: Towards 3D-Controllable Face Manipulation.
    Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung.
    ECCV 2022. [Paper] [Project]

  • GAN-Control: Explicitly Controllable GANs.
    Alon Shoshan, Nadav Bhonker, Igor Kviatkovsky, Gerard Medioni.
    ICCV 2021. [Paper] [Project] [Code]

  • CONFIG: Controllable Neural Face Image Generation.
    Marek Kowalski, Stephan J. Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, Jamie Shotton.
    ECCV 2020. [Paper] [Code]

  • DiscoFaceGAN: Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning.
    Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, Xin Tong.
    CVPR 2020. [Paper] [Code]

  • StyleRig: Rigging StyleGAN for 3D Control over Portrait Images.
    Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt.
    CVPR 2020 (oral). [Paper] [Project]

  • PIE: Portrait Image Embedding for Semantic Control.
    Ayush Tewari, Mohamed Elgharib, Mallikarjun B R., Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, Christian Theobalt.
    TOG (SIGGRAPH Asia) 2020. [Paper] [Project]

3D Prior Knowledge as Constraints

  • 3D-Aware Indoor Scene Synthesis with Depth Priors.
    Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, Qifeng Chen.
    ECCV 2022 (oral). [Paper] [Project] [Code]

  • NGP: Towards a Neural Graphics Pipeline for Controllable Image Generation.
    Xuelin Chen, Daniel Cohen-Or, Baoquan Chen, Niloy J. Mitra.
    Eurographics 2021. [Paper] [Code]

  • Lifting 2D StyleGAN for 3D-Aware Face Generation.
    Yichun Shi, Divyansh Aggarwal, Anil K. Jain.
    CVPR 2021. [Paper] [Code]

  • RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis.
    Atsuhiro Noguchi, Tatsuya Harada.
    ICLR 2020. [Paper] [Code]

  • Visual Object Networks: Image Generation with Disentangled 3D Representation.
    Jun-Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum, William T. Freeman.
    NeurIPS 2018. [Paper] [Project] [Code]

  • 3D Shape Induction from 2D Views of Multiple Objects.
    Matheus Gadelha, Subhransu Maji, Rui Wang.
    3DV 2017. [Paper] [Project] [Code]

  • Generative Image Modeling using Style and Structure Adversarial Networks.
    Xiaolong Wang, Abhinav Gupta.
    ECCV 2016. [Paper] [Project] [Code]

3D-aware GANs for a Single Image Category

Unconditional 3D Generative Models

  • BallGAN: 3D-aware Image Synthesis with a Spherical Background.
    Minjung Shin, Yunji Seo, Jeongmin Bae, Young Sun Choi, Hyunsu Kim, Hyeran Byun, Youngjung Uh.
    ICCV 2023. [Paper] [Project] [Code]

  • Mimic3D: Thriving 3D-Aware GANs via 3D-to-2D Imitation.
    Xingyu Chen, Yu Deng, Baoyuan Wang.
    ICCV 2023. [Paper] [Project]

  • GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds.
    Jianfeng Xiang, Jiaolong Yang, Yu Deng, Xin Tong.
    ICCV 2023. [Paper] [Project]

  • Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis.
    Alex Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, Koki Nagano.
    TOG (SIGGRAPH) 2023. [Paper] [Project]

  • VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids.
    Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, Andreas Geiger.
    NeurIPS 2022. [Paper] [Code]

  • GeoD: Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator.
    Zifan Shi, Yinghao Xu, Yujun Shen, Deli Zhao, Qifeng Chen, Dit-Yan Yeung.
    NeurIPS 2022. [Paper] [Project]

  • EpiGRAF: Rethinking training of 3D GANs.
    Ivan Skorokhodov, Sergey Tulyakov, Yiqun Wang, Peter Wonka.
    NeurIPS 2022. [Paper] [Project] [Code]

  • VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids.
    Schwarz, Katja, Sauer, Axel, Niemeyer, Michael, Liao, Yiyi, and Geiger, Andreas.
    NeurIPS 2022. [Paper] [Project]

  • Injecting 3D Perception of Controllable NeRF-GAN into StyleGAN for Editable Portrait Image Synthesis.
    Jeong-gi Kwak, Yuanming Li, Dongsik Yoon, Donghyeon Kim, David Han, Hanseok Ko.
    ECCV 2022. [Paper] [Project] [Code]

  • Generative Multiplane Images: Making a 2D GAN 3D-Aware.
    Xiaoming Zhao, Fangchang Ma, David Güera, Zhile Ren, Alexander G. Schwing, Alex Colburn.
    ECCV 2022. [Paper] [Project] [Code]

  • 3D-FM GAN: Towards 3D-Controllable Face Manipulation.
    Yuchen Liu, Zhixin Shu, Yijun Li, Zhe Lin, Richard Zhang, and Sun-Yuan Kung.
    ECCV 2022. [Paper] [Project]

  • EG3D: Efficient Geometry-aware 3D Generative Adversarial Networks.
    Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein.
    CVPR 2022. [Paper] [Project] [Code]

  • StylizedNeRF: Consistent 3D Scene Stylization as Stylized NeRF via 2D-3D Mutual Learning.
    Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, Lin Gao.
    CVPR 2022. [Paper]

  • Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis.
    Xuanmeng Zhang, Zhedong Zheng, Daiheng Gao, Bang Zhang, Pan Pan, Yi Yang.
    CVPR 2022. [Paper] [Code]

  • Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images.
    Ayush Tewari, Mallikarjun B R, Xingang Pan, Ohad Fried, Maneesh Agrawala, Christian Theobalt.
    CVPR 2022. [Paper] [Project]

  • GIRAFFE HD: A High-Resolution 3D-aware Generative Model.
    Yang Xue, Yuheng Li, Krishna Kumar Singh, Yong Jae Lee.
    CVPR 2022. [Paper] [Code]

  • StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation.
    Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, Ira Kemelmacher-Shlizerman.
    CVPR 2022. [Paper] [Project] [Code]

  • FENeRF: Face Editing in Neural Radiance Fields.
    Jingxiang Sun, Xuan Wang, Yong Zhang, Xiaoyu Li, Qi Zhang, Yebin Liu, Jue Wang.
    CVPR 2022. [Paper] [Code]

  • LOLNeRF: Learn from One Look.
    Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, Andrea Tagliasacchi.
    CVPR 2022. [Paper] [Project]

  • GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation.
    Yu Deng, Jiaolong Yang, Jianfeng Xiang, Xin Tong.
    CVPR 2022. [Paper] [Project] [Code]

  • VolumeGAN: 3D-aware Image Synthesis via Learning Structural and Textural Representations.
    Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, Bolei Zhou.
    CVPR 2022. [Paper] [Project] [Code]

  • Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks.
    Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, Jinwoo Shin.
    ICLR 2022. [Paper] [Project] [Code]

  • StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis.
    Jiatao Gu, Lingjie Liu, Peng Wang, Christian Theobalt.
    ICLR 2022. [Paper] [Project]

  • MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation.
    Safa C. Medin, Bernhard Egger, Anoop Cherian, Ye Wang, Joshua B. Tenenbaum, Xiaoming Liu, Tim K. Marks.
    AAAI 2022. [Paper]

  • A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis.
    Xingang Pan, Xudong Xu, Chen Change Loy, Christian Theobalt, Bo Dai.
    NeurIPS 2021. [Paper]

  • pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis.
    Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, Gordon Wetzstein.
    CVPR 2021. [Paper] [Project] [Code]

  • GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields.
    Michael Niemeyer, Andreas Geiger.
    CVPR 2021 (Best Paper). [Paper] [Project] [Code]

  • BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images.
    Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, Niloy Mitra.
    NeurIPS 2020. [Paper] [Project] [Code]

  • GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis.
    Katja Schwarz, Yiyi Liao, Michael Niemeyer, Andreas Geiger.
    NeurIPS 2020. [Paper] [Project] [Code]

  • HoloGAN: Unsupervised learning of 3D representations from natural images.
    Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, Yong-liang Yang.
    ICCV 2019. [Paper] [Project] [Code]

Conditional 3D Generative Models

3D-aware Diffusion Models for a Single Image Category

  • Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction.
    Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, Hao Su.
    ICCV 2023. [PDF] [Project] [Code]

  • 3D-aware Image Generation using 2D Diffusion Models.
    Jianfeng Xiang, Jiaolong Yang, Binbin Huang, Xin Tong.
    ICCV 2023. [Paper] [Project] [Code]

  • HoloFusion: Towards Photo-realistic 3D Generative Modeling.
    Animesh Karnewar, Niloy J. Mitra, Andrea Vedaldi, David Novotny.
    ICCV 2023. [Paper] [Project]

  • HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion.
    Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai.
    ICCV 2023. [Paper] [Project]

  • LatentSwap3D: Semantic Edits on 3D Image GANs.
    Enis Simsar, Alessio Tonioni, Evin Pınar Örnek, Federico Tombari.
    ICCV 2023 Workshop on AI3DCC. [Paper]

  • DiffusioNeRF: Regularizing Neural Radiance Fields with Denoising Diffusion Models.
    Jamie Wynn and Daniyar Turmukhambetov.
    CVPR 2023. [Paper] [Supplementary material] [COde]

  • NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models.
    Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, Sanja Fidler.
    CVPR 2023. [Paper] [Project]

  • Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion.
    Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, Baining Guo.
    CVPR 2023. [Paper] [Project]

  • DiffRF: Rendering-guided 3D Radiance Field Diffusion.
    Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Peter Kontschieder, Matthias Nießner.
    CVPR 2023 (Highlight). [Paper] [Project]

  • RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation.
    Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, Paul Guerrero.
    CVPR 2023. [Paper] [Project] [Code]

  • SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction.
    Zhizhuo Zhou, Shubham Tulsiani.
    CVPR 2023. [Paper] [Project] [Code]

  • HoloDiffusion: Training a 3D Diffusion Model using 2D Images.
    Animesh Karnewar, Andrea Vedaldi, David Novotny, Niloy Mitra.
    CVPR 2023. [Paper] [Project]

  • 3DiM: Novel View Synthesis with Diffusion Models.
    Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, Mohammad Norouzi.
    ICLR 2023. [Paper] [Project]

  • 3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models.
    Biao Zhang, Jiapeng Tang, Matthias Niessner, Peter Wonka.
    SIGGRAPH 2023. [Paper] [Project] [Code]

  • GAUDI: A Neural Architect for Immersive 3D Scene Generation.
    Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Josh Susskind.
    NeurIPS 2022. [Paper] [Project]

  • Learning a Diffusion Prior for NeRFs.
    Guandao Yang, Abhijit Kundu, Leonidas J. Guibas, Jonathan T. Barron, Ben Poole.
    arxiv 2023. [Paper]

  • Adding 3D Geometry Control to Diffusion Models.
    Wufei Ma, Qihao Liu, Jiahao Wang, Angtian Wang, Yaoyao Liu, Adam Kortylewski, Alan Yuille.
    arxiv 2023. [Paper]

  • Generative Novel View Synthesis with 3D-Aware Diffusion Models.
    Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein.
    arxiv 2023. [Paper] [Project] [Code]

  • 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models.
    Gimin Nam, Mariem Khlifi, Andrew Rodriguez, Alberto Tono, Linqi Zhou, Paul Guerrero.
    arxiv 2022. [Paper]

3D-Aware Generative Models on ImageNet

  • VQ3D: Learning a 3D-Aware Generative Model on ImageNet.
    Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, Deqing Sun.
    ICCV 2023 (Oral). [Paper] [Project]

  • 3D Generation on ImageNet.
    Ivan Skorokhodov, Aliaksandr Siarohin, Yinghao Xu, Jian Ren, Hsin-Ying Lee, Peter Wonka, Sergey Tulyakov.
    ICLR 2023 (Oral). [Paper] [Project] [Code]

3D-aware Video Synthesis

INR-based 3D Novel View Synthesis

Neural Scene Representations

Acceleration

From Constrained to In-the-wild Conditions

Few Images

  • GRF: Learning a General Radiance Field for 3D Representation and Rendering.
    Alex Trevithick, Bo Yang.
    ICCV 2021. [Paper] [Code]

  • MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo.
    Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, Hao Su.
    ICCV 2021. [Paper] [Project] [Code]

  • CodeNeRF: Disentangled Neural Radiance Fields for Object Categories.
    Wonbong Jang, Lourdes Agapito.
    ICCV 2021. [Paper] [Project] [Code]

  • pixelNeRF: Neural Radiance Fields from One or Few Images.
    Alex Yu, Vickie Ye, Matthew Tancik, Angjoo Kanazawa.
    CVPR 2021. [Paper] [Project] [Code]

  • IBRNet: Learning Multi-View Image-Based Rendering.
    Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, Thomas Funkhouser.
    CVPR 2021. [Paper] [Project] [Code]

  • NeRF-VAE: A Geometry Aware 3D Scene Generative Model.
    Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soňa Mokrá, Danilo J. Rezende.
    ICML 2021. [Paper]

Pose-free

Varying Appearance

Large-scale Scene

  • Grid-guided Neural Radiance Fields for Large Urban Scenes.
    Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin.
    CVPR 2023. [Paper] [Project]

  • S3-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint.
    Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, Kwan-Yee K. Wong.
    NeurIPS 2022. [Paper] [Project]

  • BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering.
    Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, Dahua Lin.
    ECCV 2022. [Paper] [Project]

  • Block-NeRF: Scalable Large Scene Neural View Synthesis.
    Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Henrik Kretzschmar.
    CVPR 2022. [Paper] [Project]

  • Urban Radiance Fields.
    Konstantinos Rematas, Andrew Liu, Pratul P. Srinivasan, Jonathan T. Barron, Andrea Tagliasacchi, Thomas Funkhouser, Vittorio Ferrari.
    CVPR 2022. [Paper] [Project]

  • Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs.
    Haithem Turki, Deva Ramanan, Mahadev Satyanarayanan.
    CVPR 2022. [Paper] [Code]

  • Shadow Neural Radiance Fields for Multi-view Satellite Photogrammetry.
    Dawa Derksen, Dario Izzo.
    CVPR 2021. [Paper] [Code]

Dynamic Scene

The following papers are not directly related to 3D-aware image synthesis. But it would be beneficial to pay attention to those works. For example, in our survey, inverse rendering are not classified as 3D-aware image synthesis as they are not deliberately designed for this purpose. But with the inferred intrinsic components, photorealistic images can be rendered. 3D reconstruction models geometry only with no appearance information, meaning them not able to render images with photorealistic textures. But these representations have been introduced as the geometric representation along with a textural representation (e.g., Texture Field) for 3D-aware image synthesis.

3D Representations

  • K-Planes: Explicit Radiance Fields in Space, Time, and Appearance.
    Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, Angjoo Kanazawa.
    CVPR 2023. [Paper] [Project] [Code]

  • HexPlane: A Fast Representation for Dynamic Scenes.
    Ang Cao, Justin Johnson.
    CVPR 2023. [Paper] [Project] [Code]

  • GIFS: Neural Implicit Function for General Shape Representation.
    Jianglong Ye, Yuntao Chen, Naiyan Wang, Xiaolong Wang.
    CVPR 2022. [Paper] [Project] [Code]

  • Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields.
    Wang Yifan, Lukas Rahmann, Olga Sorkine-Hornung.
    ICLR 2022. [Paper] [Project] [Code]

  • Neural Volumes: Learning Dynamic Renderable Volumes from Images.
    Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, Yaser Sheikh.
    TOG 2019. [Paper] [Code]

  • DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.
    eong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, Steven Lovegrove.
    CVPR 2019. [Paper] [Code]

  • Occupancy Networks: Learning 3D Reconstruction in Function Space.
    Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, Andreas Geiger.
    CVPR 2019. [Paper] [Project] [Code]

Neural Inverse Rendering (Neural De-rendering)

Inverse rendering is to infer underlying intrinsic components of a scene from rendered 2D images. These properties include shape (surface, depth, normal), material (albedo, reflectivity, shininess), and lighting (direction, intensity), which can be further used to render photorealistic images.

Neural Rerendering

  • Hybrid Neural Fusion for Full-frame Video Stabilization.
    Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, Jia-Bin Huang.
    ICCV 2021. [Paper] [Code]

  • Neural Lumigraph Rendering.
    Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, Gordon Wetzstein.
    CVPR 2021. [Paper] [Project] [Data]

  • Neural Re-Rendering of Humans from a Single Image.
    Kripasindhu Sarkar, Dushyant Mehta, Weipeng Xu, Vladislav Golyanik, Christian Theobalt.
    ECCV 2020. [Paper]

  • Neural Rerendering in the Wild.
    Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, Ricardo Martin-Brualla.
    CVPR 2019. [Paper]

  • Revealing Scenes by Inverting Structure from Motion Reconstructions.
    Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, Sudipta N. Sinha.
    CVPR 2019. [Paper]

Datasets

Summary of popular 3D-aware image synthesis datasets.

Multi-view image collections

The images are rendered or collected according to different experimental settings, such as Synthetic-NeRF dataset, the DTU dataset, and the Tanks and Temples dataset for general purposes, the crowded Phototourism dataset for varying lighting conditions, the Blender Forward Facing (BLEFF) dataset to benchmark camera parameter estimation and novel view synthesis quality, and the San Francisco Alamo Square Dataset for large-scale scenes.

Examples of multi-view image datasets.

dataset published in # scene # samples per scene range (m × m) resolution keyword
DeepVoxels CVPR 2019 4 simple objects 479 / 1,000 \ 512 × 512 synthetic, 360 degree
NeRF Synthetics ECCV 2020 8 complex objects 100 / 200 \ 800 ×800 synthetic, 360 degree
NeRF Captured ECCV 2020 8 complex scenes 20-62 a few 1,008 × 756 real, forward-facing
DTU CVPR 2014 124 scenes 49 or 64 a few to thousand 1,600 × 1,200 often used in few-views
Tanks and Temples CVPR 2015 14 objects and scenes 4,395 - 21,871 dozen to thousand 8-megapixel real, large-scale
Phototourism IJCV 2021 6 landmarks 763-2,000 dozen to thousand 564-1,417 megapixel varying illumination
Alamo Square CVPR 2022 San Francisco 2,818,745 570 × 960 1,200 × 900 real, large-scale

Single-view image collections

Summary of popular single-view image datasets organized by their major categories and sorted by their popularity.

dataset year category # samples resolution keyword
FFHQ CVPR 2019 Human Face 70k 1024 × 1024 single simple-shape
AFHQ CVPR 2020 Cat, Dog, and Wildlife 15k 512 × 512 single simple-shape
CompCars CVPR 2015 Real Car 136K 256 × 256 single simple-shape
CARLA CoRL 2017 Synthetic Car 10k 128 × 128 single simple-shape
CLEVR CVPR 2017 Objects 100k 256 × 256 multiple, simple-shape
LSUN 2015 Bedroom 300K 256 × 256 single, simple-shape
CelebA ICCV 2015 Human Face 200k 178 × 218 single simple-shape
CelebA-HQ ICLR 2018 Human Face 30k 1024 × 1024 single, simple-shape
MetFaces NeurIPS 2020 Art Face 1336 1024 × 1024 single, simple-shape
M-Plants NeurIPS 2022 Variable-Shape 141,824 256 × 256 single, variable-shape
M-Food NeurIPS 2022 Variable-Shape 25,472 256 × 256 single, variable-shape

Citation

If this repository benefits your research, please consider citing our paper.

  @inproceedings{xia2023survey,
    title={A Survey on Deep Generative 3D-aware Image Synthesis},
    author={Xia, Weihao and Xue, Jing-Hao},
    booktitle={ACM Computing Surveys (CSUR)},
    year={2023}
  }

License

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

About

[CSUR 2023] A Survey on Deep Generative 3D-aware Image Synthesis

weihaox.github.io/3D-aware-Gen/

Topics

computer-vision computer-graphics 3d-aware-image-synthesis

Resources

Readme

License

MIT license
Activity

Stars

156 stars

Watchers

12 watching

Forks

11 forks
Report repository

Contributors 2

  •  
  •  

Languages