Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering

Yang, Bangbang; Zhang, Yinda; Xu, Yinghao; Li, Yijin; Zhou, Han; Bao, Hujun; Zhang, Guofeng; Cui, Zhaopeng

doi:10.1109/iccv48922.2021.01352

Cited by 165 publications

(98 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our solution to the first problem is to add a parsing branch. Since the portrait parts of the same semantic category share similar motion patterns and texture information, it will be beneficial for the appearance and geometry learning in NeRF, which is also proven in recent implicit representation studies [73][74][75]81]. As shown in Fig.…”

Section: Semantic-aware Dynamic Ray Samplingmentioning

confidence: 89%

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Liu¹,

Xu²,

Wu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Animating high-fidelity video portrait with speech audio is crucial for virtual reality and digital entertainment. While most previous studies rely on accurate explicit structural information, recent works explore the implicit scene representation of Neural Radiance Fields (NeRF) for realistic generation. In order to capture the inconsistent motions as well as the semantic difference between human head and torso, some work models them via two individual sets of NeRF, leading to unnatural results. In this work, we propose Semantic-aware Speaking Portrait NeRF (SSP-NeRF), which creates delicate audio-driven portraits using one unified set of NeRF. The proposed model can handle the detailed local facial semantics and the global head-torso relationship through two semantic-aware modules. Specifically, we first propose a Semantic-Aware Dynamic Ray Sampling module with an additional parsing branch that facilitates audio-driven volume rendering. Moreover, to enable portrait rendering in one unified neural radiance field, a Torso Deformation module is designed to stabilize the large-scale non-rigid torso motions. Extensive evaluations demonstrate that our proposed approach renders more realistic video portraits compared to previous methods. Project page: https://alvinliu0.github.io/projects/SSP-NeRF.

show abstract

Section: Semantic-aware Dynamic Ray Samplingmentioning

confidence: 89%

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Liu¹,

Xu²,

Wu³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Our model also incorporates a learned camera pose refinement which has been explored in previous works [34,59,66,69,70]. Some NeRF-based methods use segmentation data to isolate and reconstruct static [67] or moving objects (such [3]. The first MLP fσ predicts the density σ for a position x in space.…”

Section: Novel View Synthesismentioning

confidence: 99%

“…Further, the inability to render scenes containing dynamic objects currently limits the applicability of Block-NeRF towards closed-loop simulation tasks in robotics. In the future, these issues could be addressed by learning transient objects during the optimization [40], or directly modeling dynamic objects [44,67]. In particular, the scene could be composed of multiple Block-NeRFs of the environment and individual controllable object NeRFs.…”

Section: Limitations and Future Workmentioning

confidence: 99%

Block-NeRF: Scalable Large Scene Neural View Synthesis

Tancik¹,

Casser²,

Yan³

et al. 2022

Preprint

View full text Add to dashboard Cite

JuneFigure 1. Block-NeRF is a method that enables large-scale scene reconstruction by representing the environment using multiple compact NeRFs that each fit into memory. At inference time, Block-NeRF seamlessly combines renderings of the relevant NeRFs for the given area.In this example, we reconstruct the Alamo Square neighborhood in San Francisco using data collected over 3 months. Block-NeRF can update individual blocks of the environment without retraining on the entire scene, as demonstrated by the construction on the right. Video results can be found on the project website waymo.com/research/block-nerf.

show abstract

“…Recently, several works [3,4,6,20,43,49] have extended NeRF and achieved impressive progress in decomposing the renderings of a scene into semantically meaningful components, including geometry, reflectance, material, and lighting, enabling a flexible interaction with any of these components, e.g. relighting and swapping the background.…”

Section: Introductionmentioning

confidence: 99%

NeROIC: Neural Rendering of Objects from Online Image Collections

Kuang¹,

Olszewski²,

Chai³

et al. 2022

Preprint

View full text Add to dashboard Cite

We present a novel method to acquire object representations from online image collections, capturing high-quality geometry and material properties of arbitrary objects from photographs with varying cameras, illumination, and backgrounds. This enables various object-centric rendering applications such as novel-view synthesis, relighting, and harmonized background composition from challenging in-thewild input. Using a multi-stage approach extending neural radiance fields, we first infer the surface geometry and refine the coarsely estimated initial camera parameters, while leveraging coarse foreground object masks to improve the training efficiency and geometry quality. We also introduce a robust normal estimation technique which eliminates the effect of geometric noise while retaining crucial details. Lastly, we extract surface material properties and ambient illumination, represented in spherical harmonics with extensions that handle transient elements, e.g. sharp shadows. The union of these components results in a highly modular and efficient object acquisition framework. Extensive evaluations and comparisons demonstrate the advantages of our approach in capturing high-quality geometry and appearance properties useful for rendering applications. Our code will be released at https://formyfamily.github. io/NeROIC/ * This work was performed while the author was an intern at Snap, Inc. Online Images NeROIC OutputsNovel View Synthesis Relighting Composition

show abstract

Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering

Cited by 165 publications

References 17 publications

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation

Block-NeRF: Scalable Large Scene Neural View Synthesis

NeROIC: Neural Rendering of Objects from Online Image Collections

Contact Info

Product

Resources

About