GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Abstract

While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxelbased representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.

Point-NeRF: Point-based Neural Radiance Fields

“…They have been extended to achieve dynamic scene capture [27,39], relighting [3,5], appearance editing [57], fast rendering [16,61], and generative models [7,36,45]. However, most methods [3,27,39,57] still follow the original NeRF framework and train per-scene MLPs to represent radiance fields.…”

CG-NeRF: Conditional Generative Neural Radiance Fields

“…Appearance mapping network where ξ is the camera pose for calculating the 3D coordinate x and the viewing direction d (Schwarz et al 2020). Below, we describe the model structure designed for CG-NeRF in detail.…”

“…While these techniques utilize NeRF only for synthesizing an unseen view, recent studies have emerged that generate photorealistic multi-view images based on generative adversarial networks(GANs) (Schwarz et al 2020;Niemeyer and Geiger 2021c;Chan et al 2021). Compared to the existing 2D-based generative models, these studies can produce 3D-aware images by generating view-consistent images for given camera poses.…”

CoordX: Accelerating Implicit Neural Representation with a Split MLP Architecture

“…Second, the fully differentiable learning process allows the MLP to learn complex signals from sparsely available data points to reconstruct high-quality images or objects. Recent research has demonstrated the effectiveness of coordMLPs for a wide range of signal fitting or learning tasks such as image super-resolution (Chen et al, 2021), 3D shape representation (Park et al, 2019;Mescheder et al, 2019;Gropp et al, 2020), novel view synthesis Schwarz et al, 2020) and photo-realistic 3D scene editing (Niemeyer & Geiger, 2021). The search for more accurate and generalizable implicit neural network architectures and methodologies is an active area of research.…”

“…To address this, proposes a positional encoding (PE) method that maps the original input coordinates into a high dimensional space using a series of high frequency functions to better capture high frequency variations. A common way to enrich the representation power of these models is to add latent codes/features as additional inputs to the coordMLPs (Schwarz et al, 2020;Park et al, 2021;Chen et al, 2021;Mehta et al, 2021). These additional inputs are combined with positional coordinates and are essentially treated as additional dimensions in the input coordinates.…”