CG-NeRF: Conditional Generative Neural Radiance Fields

Jo, Kyungmin; Shim, Gyumin; Jung, Sung-Eun; Choo, Jaegul

doi:10.48550/arxiv.2112.03517

Cited by 4 publications

(6 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to address this issue, many works have employed the method of embedding explicit control into the generation process. For example, CG-NeRF [Jo et al 2021] introduces various soft conditions, including sketches as input. FENeRF involves semantic masks in the generation process as output.…”

Section: D-aware Neural Face Image Synthesismentioning

confidence: 99%

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

Jiang

Chen

Liu

et al. 2022

SIGGRAPH Asia 2022 Conference Papers

View full text Add to dashboard Cite

Figure 1: Our NeRFFaceEditing method allows users to intuitively edit a facial volume to manipulate its geometry and appearance guided by rendered semantic masks. Given an input sample (a), our method disentangles its geometry and appearance, and allows for one-or multi-label editing. We show a range of flexible face editing tasks that can be achieved with our unified framework: (b) changing the appearance according to a given reference sample while retaining the geometry and 3D consistency; (c) changing the appearance for different views with different reference samples while retaining the geometry; (d) editing multiple labels of the semantic mask for a certain view while keeping the appearance and 3D consistency; (e) editing both the geometry and appearance. The inputs used to control the appearance and geometry are highlighted in green and orange boxes, respectively.

show abstract

Section: D-aware Neural Face Image Synthesismentioning

confidence: 99%

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

Jiang

Chen

Liu

et al. 2022

SIGGRAPH Asia 2022 Conference Papers

View full text Add to dashboard Cite

show abstract

“…CG-NeRF is capable of producing 3D-aware output images that are faithful to the corresponding condition inputs. The image is from [184].…”

Section: Generative Nerfmentioning

confidence: 99%

“…Based on these prior explorations of unconditional generative NeRF, building generative NeRF conditioned on certain guidance has attracted increasing interest. Recently, Jo et al [184] propose a conditional generative neural radiance fields (CG-NeRF), which can generate multi-view images reflecting extra input conditions (e.g., images or texts) as shown in Fig. 15.…”

Section: Generative Nerfmentioning

confidence: 99%

“…Current generative NeRF models (e.g., StyleNeRF, EG3D) have enabled to model scenes with simple geometry (e.g., faces, cars) from a collection of unposed 2D images, just like the training of unconditional GANs (e.g., StyleGAN). Powered by these efforts, some 3D-aware MISE tasks have been explored, e.g., text-to-NeRF [184] and semantic-to-NeRF [193]. However, current generative NeRFs still struggle on datasets with complex geometry variation, e.g., Deep-Fashion [215] and ImageNet [137].…”

Section: Towards 3d Awarenessmentioning

confidence: 99%

See 1 more Smart Citation

Expired

Zhan¹

2022

Preprint

View full text Add to dashboard Cite

As information exists in various modalities in real world, effective interaction and fusion among multimodal information plays a key role for the creation and perception of multimodal data in computer vision and deep learning research. With superb power in modelling the interaction among multimodal information, multimodal image synthesis and editing have become a hot research topic in recent years. Different from traditional visual guidance which provides explicit clues, multimodal guidance offers intuitive and flexible means in image synthesis and editing. On the other hand, this field is also facing several challenges in alignment of features with inherent modality gaps, synthesis of high-resolution images, faithful evaluation metrics, etc. In this survey, we comprehensively contextualize the advance of the recent multimodal image synthesis \& editing and formulate taxonomies according to data modality and model architectures. We start with an introduction to different types of guidance modalities in image synthesis and editing. We then describe multimodal image synthesis and editing approaches extensively with detailed frameworks including Generative Adversarial Networks (GANs), GAN Inversion, Transformers, and other methods such as NeRF and Diffusion models. This is followed by a comprehensive description of benchmark datasets and corresponding evaluation metrics as widely adopted in multimodal image synthesis and editing, as well as detailed comparisons of different synthesis methods with analysis of respective advantages and limitations. Finally, we provide insights into the current research challenges and possible future research directions. We hope this survey could lay a sound and valuable foundation for future development of multimodal image synthesis and editing. A project associated with this survey is available at \href{https://github.com/fnzhan/MISE}{https://github.com/fnzhan/MISE}.

show abstract

“…People address these scaling issues of NeRF-based GANs in different ways, but the dominating approach is to train a separate 2D decoder to produce a high-resolution image from a low-resolution image or feature grid rendered from a NeRF backbone [43]. During the past six months, there appeared more than a dozen of methods which follow this paradigm (e.g., [6,15,71,47,79,35,75,23,72,78,64]). While using the upsampler allows to scale the model to high resolution, it comes with two severe limitations: 1) it breaks multi-view consistency of a generated object, i.e.…”

Section: Introductionmentioning

confidence: 99%

EpiGRAF: Rethinking training of 3D GANs

Skorokhodov¹,

Tulyakov²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns the geometry in a low fidelity. In this work, we show that it is possible to obtain a high-resolution 3D generator with SotA image quality by following a completely different route of simply training the model patch-wise. We revisit and improve this optimization scheme in two ways. First, we design a location-and scale-aware discriminator to work on patches of different proportions and spatial positions. Second, we modify the patch sampling strategy based on an annealed beta distribution to stabilize training and accelerate the convergence. The resulted model, named EpiGRAF, is an efficient, high-resolution, pure 3D generator, and we test it on four datasets (two introduced in this work) at 256 2 and 512 2 resolutions. It obtains state-of-the-art image quality, high-fidelity geometry and trains ≈2.5× faster than the upsampler-based counterparts.

show abstract

CG-NeRF: Conditional Generative Neural Radiance Fields

Abstract: the proposed method maintains consistent image quality on various condition types and achieves superior fidelity and diversity compared to existing NeRF-based generative models.

Cited by 4 publications

References 18 publications

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields

Expired

EpiGRAF: Rethinking training of 3D GANs

Contact Info

Product

Resources

About