News: We provide a survey, Deep Generative Models on 3D Representations: A Survey, to help the community track the evolution of this field.
Deep Generative Models on 3D Representations: A Survey
Zifan Shi*, Sida Peng*, Yinghao Xu*, Yiyi Liao, Yujun Shen
https://arxiv.org/abs/2210.15663
(* denotes equal contribution)
This repository collects the studies on 3D generation, including both 3D shape generation and 3D-aware image generation. Different from 3D reconstruction, which focuses on per-instance recovery (i.e., the data already exists in the real world), 3D generation targets learning the real distribution and hence allows sampling new data.
Overall, the paper collection is organized as follows. If you find some work is missing, feel free to raise an issue or create a pull request. We appreciate contributions in any form.
We categorize the studies on 3D shape generation according to the representation used.
- Learning Representations and Generative Models for 3D Point Clouds
ICML 2018 / Code - Multiresolution Tree Networks for 3D Point Cloud Processing
ECCV 2018 / Code / Project Page - 3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions
ICCV 2019 / Code - Point Cloud GAN
ICLR 2019 / Code - Learning Localized Generative Models for 3D Point Clouds via Graph Convolution
ICLR 2019 / Code - PointFlow : 3D Point Cloud Generation with Continuous Normalizing Flows
ICCV 2019 / Code - Spectral-GANs for High-Resolution 3D Point-Cloud Generation
IROS 2020 / Code - Progressive Point Cloud Deconvolution Generation Network
ECCV 2020 / Code - A Progressive Conditional Generative Adversarial Network for Generating Dense and Colored 3D Point Clouds
3DV 2020 / Code - Adversarial Autoencoders for Generating 3D Point Clouds
ICLR 2020 / Code - Learning Gradient Fields for Shape Generation
ECCV 2020 / Code / Project Page - SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds
NeurIPS 2020 / Code - Discrete Point Flow Networks for Efficient Point Cloud Generation
ECCV 2020 / Code - Pointgrow: Autoregressively Learned Point Cloud Generation with Self-Attention
WACV 2020 / Code / Project Page - MRGAN: MultiRooted 3D Shape Generation with Unsupervised Part Disentanglement
ICCVW 2021 - SP-GAN: Sphere-Guided 3D Shape Generation and Manipulation
SIGGRAPH 2021 / Code - Generative PointNet: Deep Energy-Based Learning on Unordered Point Sets for
3D Generation, Reconstruction and Classification
CVPR 2021 / Code / Project Page - Diffusion Probabilistic Models for 3D Point Cloud Generation
CVPR 2021 / Code - 3D Shape Generation and Completion through Point-Voxel Diffusion
ICCV 2021 / Code / Project Page - ManiFlow: Implicitly Representing Manifolds with Normalizing Flows
3DV 2022 - LION: Latent Point Diffusion Models for 3D Shape Generation
NeurIPS 2022 / Code / Project Page
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
NeurIPS 2016 / Code / Project Page - Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
arXiv 2016 / Code - SAGNet: Structure-aware Generative Network for 3D-Shape Modeling
SIGGRAPH 2019 / Code / Project Page - Generalized Autoencoder for Volumetric Shape Generation
CVPRW 2020 / Code - PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes
CVPR 2020 / Code - Learning Part Generation and Assembly for Structure-Aware Shape Synthesis
AAAI 2020 - Generative VoxelNet: Learning Energy-Based Models for 3D Shape Synthesis and Analysis
TPAMI 2020 / Code / Project Page - Octree Transformer: Autoregressive 3D Shape Generation on Hierarchically Structured Sequences
arXiv 2021 - AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation
CVPR 2022 / Project Page / Code
- SDM-Net: Deep Generative Network for Structured Deformable Mesh
SIGGRAPH Asia 2019 / Project Page - PolyGen: An Autoregressive Generative Model of 3D Meshes
ICML 2020 / Code - TM-NET: Deep Generative Networks for Textured Meshes
TOG 2021 / Code / Project Page - Multi-Chart Generative Surface Modeling
SIGGRAPH Asia 2018 / Code
- Learning Implicit Fields for Generative Shape Modeling
CVPR 2019 / Code / Project Page - Adversarial Generation of Continuous Implicit Shape Representations
EG 2020 / Code - DualSDF: Semantic Shape Manipulation using a Two-Level Representation
CVPR 2020 / Code / Project Page - Physically-Aware Generative Network for 3D Shape Modeling
CVPR 2021 - SurfGen: Adversarial 3D Shape Synthesis with Explicit Surface Discriminators
ICCV 2021 - 3D Shape Generation with Grid-Based Implicit Functions
CVPR 2021 / Code - Deformed Implicit Field: Modeling 3D shapes with Learned Dense Correspondence
CVPR 2021 / Code - gDNA: Towards Generative Detailed Neural Avatars
CVPR 2022 / Code / Project Page - ShapeCrafter: A Recursive Text-Conditioned 3D Shape Generation Model
arXiv 2022 - Learning to Generate 3D Shapes from a Single Example
arXiv 2022 / Code / Project Page - 3D-LDM: Neural Implicit 3D Shape Generation with Latent Diffusion Models
arXiv 2022
- ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis
SIGGRAPH Asia 2020 / Code / Project Page
We categorize the studies on 3D-aware image generation according to the representation used.
- Visual Object Networks: Image Generation with Disentangled 3D Representation
NeurIPS 2018 / Code / Project Page - HoloGAN: Unsupervised Learning of 3D representations from Natural Images
ICCV 2019 / Code - BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
NeurIPS 2020 / Code - Towards a Neural Graphics Pipeline for Controllable Image Generation
Computer Graphics Forum 2021 / Project Page - VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids
NeurIPS 2022
- Generative Image Modeling using Style and Structure Adversarial Networks
ECCV 2016 / Code - RGBD-GAN: Unsupervised 3D Representation Learning From Natural Image Datasets via RGBD Image Synthesis
ICLR 2020 / Code - 3D-Aware Indoor Scene Synthesis with Depth Priors
ECCV 2022 / Code / Project Page
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
NeurIPS 2020 / Code / Project Page - NeRF-VAE: A Geometry Aware 3D Scene Generative Model
ICML 2021 - GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
CVPR 2021 / Code / Project Page - pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis
CVPR 2021 / Code / Project Page - Unconstrained Scene Generation with Locally Conditioned Radiance Fields
ICCV 2021 / Code / Project Page - A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis
NeurIPS 2021 / Code / Project Page - Campari: Camera-aware Decomposed Generative Neural Radiance Fields
3DV 2021 / Code - CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
arXiv 2021 / Code - GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
ICCV 2021 / Code / Project Page - Generative Occupancy Fields for 3D Surface-Aware Image Synthesis
NeurIPS 2021 / Code / Project Page - 3D-Aware Semantic-Guided Generative Model for Human Synthesis
ECCV 2022 / Code - FENeRF: Face Editing in Neural Radiance Fields
CVPR 2022 / Code / Project Page - StyleNeRF: A Style-Based 3D-Aware Generator for High-resolution Image Synthesis
ICLR 2022 / Code / Project Page - StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation
CVPR 2022 / Code / Project Page - GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation
CVPR 2022 / Code / Project Page - A Generative Model for 3D Face Synthesis with HDRI Relighting
arXiv 2022 / Code / Project Page - Pix2NeRF: Unsupervised Conditional π-GAN for Single Image to Neural Radiance Fields Translation
CVPR 2022 / Code - 3D-GIF: 3D-Controllable Object Generation via Implicit Factorized Representations
arXiv 2022 - GIRAFFE HD: A High-Resolution 3D-aware Generative Model
CVPR 2022 / Code / Project Page - Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis
CVPR 2022 / Code - Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Images
CVPR 2022 / Project Page - Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations
ECCV 2022 / Code / Project Page - IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis
SIGGRAPH Asia 2022 / Code / Project Page - GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifolds
arXiv 2022 / Project Page - EpiGRAF: Rethinking training of 3D GANs
NeurIPS 2022 / Code / Project Page - AvatarGen: a 3D Generative Model for Animatable Human Avatars
arXiv 2022 / Code - cGOF: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
NeurIPS 2022 / Code / Project Page - Improving 3D-aware Image Synthesis with A Geometry-aware Discriminator
NeurIPS 2022 / Code / Project Page - GAUDI: A Neural Architect for Immersive 3D Scene Generation
NeurIPS 2022 / Code - DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis
arXiv 2022 / Code / Project Page - Learning 3D-aware Image Synthesis with Unknown Pose Distribution
arXiv 2023 / Code / Project Page - Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
CVPR 2023 / Code / Project Page
- 3D-aware Image Synthesis via Learning Structural and Textural Representations
CVPR 2022 / Code / Project Page - Efficient Geometry-aware 3D Generative Adversarial Networks
CVPR 2022 / Code / Project Page - Generative Neural Articulated Radiance Fields
NeurIPS 2022 / Project Page - GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
NeurIPS 2022 / Code / Project Page - DiffRF: Rendering-Guided 3D Radiance Field Diffusion
arXiv 2022 / Project Page - RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
arXiv 2022 / Code - Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion
arXiv 2022 / Project Page - 3D Neural Field Generation using Triplane Diffusion
arXiv 2022 / Code / Project Page
Besides explicitly learning a 3D generative model, there are also some attempts working on the 3D controllability of 2D models.
- Interpreting the latent space of gans for semantic face editing
CVPR 2020 / Code / Project Page - StyleRig: Rigging StyleGAN for 3D Control over Portrait Images
CVPR 2020 / Project Page - Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
CVPR 2020 / Code - Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis
CVPR 2020 / Code - DiscoFaceGAN: Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning
CVPR 2020 / Code - FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
SIGGRAPH Asia 2021 / Code / Project Page - Do 2D GANs Know 3D Shape? Unsupervised 3D Shape Reconstruction from 2D Image GANs
ICLR 2021 / Code / Project Page - Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
IJCV 2021 / Code / Project Page - Cross-Domain and Disentangled Face Manipulation with 3D Guidance
TVCG 2021 / Code / Project Page - Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
ICLR 2021 / Project Page - Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN
TOG 2021 / Code / Project Page - SofGAN: A Portrait Image Generator with Dynamic Styling
TOG 2021 / Code / Project Page - MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
AAAI 2022 - HeadNeRF: A Real-time NeRF-based Parametric Head Model
CVPR 2022 / Code / Project Page - GAN2X: Non-Lambertian Inverse Rendering of Image GANs
3DV 2022 / Project Page - DreamFusion: Text-to-3D using 2D Diffusion
arXiv 2022 / Project Page - Magic3D: High-Resolution Text-to-3D Content Creation
arXiv 2022 / Project Page - 3D-TOGO: Towards Text-Guided Cross-Category 3D Object Generation
AAAI 2023 - Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation
arXiv 2022 / Code / Project Page