GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation

A Survey on 3D-aware Image Synthesis

“…An example of synthetic data rendering process is to center a scan (e.g., objects from ScanNet [66], ShapeNet [67], or DeepVoxels [14]) at the origin, scale it to lie within the unit cube, and render images at sampled viewpoints. The training set can be obtained by 2019 2020 2021 2022 2023 S 2 -GAN [12] PrGAN [13] DeepVoxel [14] VON [15] HoloGAN [16] SRN [17] GANSteerability [18] RGBD-GAN [19] DVR [7] BlockGAN [20] GANLatentDiscovery [21] NeRF [5] StyleRig [22] CONFIG [23] GANSpace [24]) InterFaceGAN [25]) NGP [26] SeFa [27] GRAF [28] NeRF-W [29] PIE [30] NeRF++ [31] GIRAFFE [32] pi-GAN [2] PixelNeRF [33] GAN-Control [34] NeRF- [35] KiloNeRF [36] Mip-NeRF [37] FastNeRF [38] CAMPARI [39] BARF [40] VariTex [41] Liao et al [42] ShadeGAN [43] CIPS-3D [44] StyleNeRF [1] GOF [45] LOLNeRF [46] URF [47] EG3D [48] GRAM [49] StyleSDF ...…”

“…Real Car 136K 256×256 [1], [16], [20], [51] single simple-shape CARLA [79] CoRL 2017 Synthetic Car 10k 128×128 [2], [49], [51], [58], [71] single simple-shape CLEVRn [81] CVPR 2017 Objects 100k 256×256 [20], [32] multiple, simple-shape LSUN [80] 2015 Bedroom 300K 256 × 256 [16], [51] single, simple-shape CelebA [76] ICCV 2015 Human Face 200k 178×218 [16], [51] single simple-shape CelebA-HQ [77] ICLR 2018…”

“…HoloGAN [16] ICCV 2019 face, cat, car, LSUN camera pose unsupervised deep voxel Liao et al [42] CVPR 2020 multiple object data unconditional unsupervised 3D primitives BlockGAN [20] NeurIPS 2020 multiple object data camera pose unsupervised deep voxel GRAF [28] NeurIPS 2020 rendered chair, face, cat, bird camera matrix, pose unsupervised NeRF pi-GAN [2] CVPR 2021 face, car, CARLA camera position unsupervised NeRF GIRAFFE [32] CVPR 2021 chair, cat, face, car, church camera pose unsupervised compositional NeRF GOF [45] NeurIPS 2021 cat, car, face 3D location, camera pose unsupervised generative occupancy fields ShadeGAN [43] NeurIPS 2021 cat, face 3D location, camera pose unsupervised light field CAMPARI [39] 3DV 2021 cat, car, face, chair camera pose unsupervised decomposed NeRF StyleNeRF [1] ICLR 2022 face, cat, car camera pose unsupervised NeRF GRAM [49] CVPR 2022 cat, face, CARLA camera pose unsupervised EG3D [48] CVPR 2022 cat, face camera parameters unsupervised tri-plane 3D representation VolumeGAN [51] CVPR 2022 cat, car, face, bedroom, CARLA camera pose unsupervised StyleSDF [50] CVPR 2022 cat, face camera pose unsupervised SDF Pix2NeRF [71] CVPR 2022 face, CARLA, rendered image image unsupervised Sem2NeRF [57] ECCV 2022 cat, face semantic mask, camera pose semantic mask and image SURF-GAN [63] ECCV 2022 face camera pose unsupervised EpiGRAF [60] NeurIPS 2022 cat, face, variable-shape camera pose unsupervised tri-plane 3D representation IDE-3D [61] TOG 2022 face camera pose semantic mask and image tri-plane 3D representation control over the 3D parameters θ gives the edited result x . Here, θ could be human-interpretable attribute descriptions or a set of parameters from 3D models.…”

“…Due to the high memory and computation cost of volume rendering, direct rendering at target resolution is not efficient and struggles to generate images at high-resolution (512×512 and beyond). Some recently-developed methods make use of a two-stage rendering process [1], [32], [48], [50], [51] or develop efficient volume rendering strategy [43], [44], [49] for high-resolution image generation. Meanwhile, aimed to reduce view-inconsistent artifacts brought by the 2D renderers, they adopt different strategies such as NeRF path regularization [1] and dual discriminators [48].…”

“…VoxGARF [59] accelerates the rendering from the perspective of spatial sparsity, where volume rendering yields a foreground image and an alpha mask. The second group aims to develop efficient sampling strategy [43], [45], [49]. They constrain the point sampling in a reduced space rather than anywhere in the volume.…”