Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Henzler, Philipp; Mitra, Niloy J.; Ritschel, Tobias

doi:10.1109/iccv.2019.01008

Cited by 156 publications

(135 citation statements)

References 17 publications

Supporting

Mentioning

128

Contrasting

Order By: Relevance

“…This allows rendering photo-realistic images from different viewpoints. Early methods use GAN-based architectures for building 3D voxel [69,11,39,19] or mesh [55,18] representations. However, they mostly focus on learning un-textured 3D structures.…”

Section: Related Workmentioning

confidence: 99%

3D-Aware Semantic-Guided Generative Model for Human Synthesis

Zhang¹,

Sangineto²,

Tang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Generative Neural Radiance Field (GNeRF) models, which extract implicit 3D representations from 2D images, have recently been shown to produce realistic images representing rigid objects, such as human faces or cars. However, they usually struggle to generate high-quality images representing non-rigid objects, such as the human body, which is of a great interest for many computer graphics applications. This paper proposes a 3D-aware Semantic-Guided Generative Model (3D-SGAN) for human image synthesis, which integrates a GNeRF and a texture generator. The former learns an implicit 3D representation of the human body and outputs a set of 2D semantic segmentation masks. The latter transforms these semantic masks into a real image, adding a realistic texture to the human appearance. Without requiring additional 3D information, our model can learn 3D human representations with a photorealistic controllable generation. Our experiments on the DeepFashion dataset show that 3D-SGAN significantly outperforms the most recent baselines.

show abstract

Section: Related Workmentioning

confidence: 99%

3D-Aware Semantic-Guided Generative Model for Human Synthesis

Zhang¹,

Sangineto²,

Tang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…tomographic rendering step, while other work has shown how it can be differentiated directly [Henzler et al 2019;Lombardi et al 2019;Mildenhall et al 2020]. Our approach avoids tomography and works with differentiable warping [Jaderberg et al 2015] with consistency handling inspired by unsupervised depth reconstruction [Godard et al 2017;Zhou et al 2017].…”

Section: Scenementioning

confidence: 99%

X-Fields

et al. 2020

Self Cite

View full text Add to dashboard Cite

We suggest to represent an X-Field ---a set of 2D images taken across different view, time or illumination conditions, i.e., video, lightfield, reflectance fields or combinations thereof---by learning a neural network (NN) to map their view, time or light coordinates to 2D images. Executing this NN at new coordinates results in joint view, time or light interpolation. The key idea to make this workable is a NN that already knows the "basic tricks" of graphics (lighting, 3D projection, occlusion) in a hard-coded and differentiable form. The NN represents the input to that rendering as an implicit map, that for any view, time, or light coordinate and for any pixel can quantify how it will move if view, time or light coordinates change (Jacobian of pixel position with respect to view, time, illumination, etc.). Our X-Field representation is trained for one scene within minutes, leading to a compact set of trainable parameters and hence real-time navigation in view, time and illumination.

show abstract

“…Learning 3D-aware image generation with Generative Adversarial Networks (GAN) [20] has attracted a surge of attention in recent years [11,13,15,24,34,[44][45][46]57]. Given an unstructured 2D image collection, GANs are trained to synthesize geometrically-consistent multiview imagery of novel instances.…”

Section: Introductionmentioning

confidence: 99%

“…Given an unstructured 2D image collection, GANs are trained to synthesize geometrically-consistent multiview imagery of novel instances. In particular, methods [11,24,57] that use the volumetric rendering paradigm [18,27] to composite an output image have demonstrated impressive results with more "strict" 3D consistency by virtue of an explicit, physics-based rendering process.…”

Section: Introductionmentioning

confidence: 99%

“…One prominent hurdle is the high computation and memory requirements for training a volumetric representation. Methods [11,57] that use neural radiance field (NeRF) [42] generators can greatly reduce the complexity of voxel-based approaches [24], but the volume integrations approximated by sampling points along viewing rays are still costly for both training and inference. This problem becomes even more pronounced in GAN training where a full image (rather than sparse pixels) needs to be rendered to train the discriminator.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

Yu¹,

Yang²,

Xiang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Figure 1. Image samples randomly generated by our method (256×256 resolution). Trained on unstructured image collections (FFHQ [28] and Cats [70] in this figure), our method can generate view-controllable images that are of high quality (e.g., see the fine details) and strong 3D consistency (e.g., see the correct parallax when view changes). (The second row contains animations best viewed in Adobe Reader; more can be found on the project page

show abstract

Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Cited by 156 publications

References 17 publications

3D-Aware Semantic-Guided Generative Model for Human Synthesis

3D-Aware Semantic-Guided Generative Model for Human Synthesis

X-Fields

GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

Contact Info

Product

Resources

About