Space-time Neural Irradiance Fields for Free-Viewpoint Video

Xian, Wenqi; Huang, Jia-Bin; Kopf, Johannes; Kim, Chang-Il

doi:10.48550/arxiv.2011.12950

Cited by 20 publications

(26 citation statements)

References 79 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these neural representations above can only handle static scenes, and the literature on dynamic scene neural representation remains sparse. Recent work Ost et al 2020;Pumarola et al 2020;Rebain et al 2020;Tretschk et al 2020;Xian et al 2020] extend the approach NeRF [Mildenhall et al 2020a] using neural radiance field into the dynamic setting. They decompose the task into learning a spatial mapping from the canonical scene to the current scene at each time step and regressing the canonical radiance field.…”

Section: Related Workmentioning

confidence: 99%

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

Generating free-viewpoint videos is critical for immersive VR/AR experience, but recent neural advances still lack the editing ability to manipulate the visual perception for large dynamic scenes. To fill this gap, in this paper, we propose the first approach for editable free-viewpoint video generation for large-scale view-dependent dynamic scenes using only 16 cameras. The core of our approach is a new layered neural representation, where each dynamic entity, including the environment itself, is formulated into a spatiotemporal coherent neural layered radiance representation called ST-NeRF. Such a layered representation supports manipulations of the dynamic scene while still supporting a wide free viewing experience. In our ST-NeRF, we represent the dynamic entity/layer as a continuous function, which achieves the disentanglement of location, deformation as well as the appearance of the dynamic entity in a continuous and self-supervised manner. We propose a scene parsing 4D label map tracking to disentangle the spatial information explicitly and a continuous deform module to disentangle the temporal motion implicitly. An object-aware volume rendering scheme is further

show abstract

Section: Related Workmentioning

confidence: 99%

Editable free-viewpoint video using a layered neural representation

Zhang

Liu

et al. 2021

ACM Trans. Graph.

View full text Add to dashboard Cite

show abstract

“…For neural modeling and rendering of dynamic scenes, NHR [64] embeds spatial features into sparse dynamic point-clouds, Neural Volumes [30] transforms input images into a 3D volume representation by a VAE network. More recently, [26,44,47,48,61,65,74] extend neural radiance field (NeRF) [36] into the dynamic setting. They learn a spatial mapping from the canonical scene to the current scene at each time step and regress the canonical radiance field.…”

Section: Blended Imagementioning

confidence: 99%

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Sun

Chen

et al. 2021

Proceedings of the 29th ACM International Conference on Multimedia

View full text Add to dashboard Cite

human-object rendering scheme, which combines direction-aware neural blending weight learning and spatial-temporal texture completion to provide high-resolution and photo-realistic texture results in the occluded scenarios. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and texture reconstruction in free viewpoints for challenging human-object interactions. CCS CONCEPTS• Computing methodologies → Image-based rendering.

show abstract

“…To regularize the training, Neural Body [33] combines NeRF with a deformable human body model (e.g., SMPL [26]). Despite the promising results, these general NeRF [19,53] and human-specific NeRF [13,32,33,35,50] methods must be optimized for each new video separately, and generalize poorly on unseen scenarios. Generalizable NeRFs [36,47,52] try to avoid the expensive per-scene optimization by imageconditioning using pixel-aligned features.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, neural radiance fields (NeRF) [28,13,19,32,33,35,36,47,50,52,53] have shown photo-realistic novel view synthesis results in per-scene optimization settings. To avoid the expensive per-scene training and improve the practicality, generalizable NeRFs [36,52,47] have been proposed which use image-conditioned, pixel-aligned features and achieve feed-forward view synthesis from sparse input views [36,52].…”

Section: Introductionmentioning

confidence: 99%

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

Kwon,

Kim,

Ceylan

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we aim at synthesizing a free-viewpoint video of an arbitrary human performance using sparse multi-view cameras. Recently, several works have addressed this problem by learning person-specific neural radiance fields (NeRF) to capture the appearance of a particular human. In parallel, some work proposed to use pixel-aligned features to generalize radiance fields to arbitrary new scenes and objects. Adopting such generalization approaches to humans, however, is highly challenging due to the heavy occlusions and dynamic articulations of body parts. To tackle this, we propose Neural Human Performer, a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Specifically, we first introduce a temporal transformer that aggregates tracked visual features based on the skeletal body motion over time. Moreover, a multi-view transformer is proposed to perform cross-attention between the temporally-fused features and the pixel-aligned features at each time step to integrate observations on the fly from multiple views. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses. The video results and code are available at https://youngjoongunc.github.io/nhp.Preprint. Under review.

show abstract

Space-time Neural Irradiance Fields for Free-Viewpoint Video

Cited by 20 publications

References 79 publications

Editable free-viewpoint video using a layered neural representation

Editable free-viewpoint video using a layered neural representation

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering

Contact Info

Product

Resources

About