Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

Yu, Tao; Zheng, Zerong; Guo, Kaiwen; Liu, Pengpeng; Dai, Qionghai; Liu, Yebin

doi:10.1109/cvpr46437.2021.00569

Cited by 195 publications

(96 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…UnstructuredFusion [57] achieves an unstructured multi-view setup. Function4D [60] combines temporal volumetric fusion and deep implicit functions to generate complete capture even with severe topological changes. However, these methods either cannot handle modeling human-object interactions or generate photorealistic rendering results.…”

Section: Related Workmentioning

confidence: 99%

“…Neural networks based on implicit functions are good at complete reconstruction but lack geometry details and temporal consistency, while traditional volumetric capture methods [8,32,57] have enabled the temporal-consistent reconstruction results. From the insight of this complementarity, we follow Function4d [60] to combine the non-rigid fusion with implicit functions by extracting features from TSDF volume. However, when facing severe occlusions in human-object interactions, [60] will fail since their non-rigid fusion in a sliding way could not provide complete TSDF features.…”

Section: Neural Human Reconstructionmentioning

confidence: 99%

“…From the insight of this complementarity, we follow Function4d [60] to combine the non-rigid fusion with implicit functions by extracting features from TSDF volume. However, when facing severe occlusions in human-object interactions, [60] will fail since their non-rigid fusion in a sliding way could not provide complete TSDF features. Therefore, we non-rigidly fuse a key volume V k as a reference model, in which we track the motion field from the live frame to it and integrate each depth into this TSDF volume.…”

Section: Neural Human Reconstructionmentioning

confidence: 99%

“…Recent learning-based techniques enable robust human modeling from only light-weight inputs. In particular, various approaches [38,39,47] utilize implicit function to model human geometry, which is also widely adopted in the volumetric capture pipeline [24,25,43,60]. But these methods are restricted to only human without modeling humanobject interactions, let alone generating compelling photorealistic texture.…”

Section: Introductionmentioning

confidence: 99%

“…For human reconstruction, we propose a fusion-based neural implicit scheme to reason about the human-only geometry details in novel views. Specifically, it combines pixel-aligned features with an occlusion-aware truncated projective SDF (TSDF) feature [60], by utilizing a traditional key-volume non-rigid fusion pipeline [8,55] in a human-only manner. Such a key-volume fusion-based implicit scheme handles topology changes and occlusions effectively.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Jiang¹,

Jiang²,

Sun³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

4D modeling of human-object interactions is critical for numerous applications. However, efficient volumetric capture and rendering of complex interaction scenarios, especially from sparse inputs, remain challenging. In this paper, we propose NeuralFusion, a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. It marries traditional non-rigid fusion with recent neural implicit modeling and blending advances, where the captured humans and objects are layerwise disentangled. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion, as well as a template-aid robust object tracking pipeline. Our scheme enables detailed and complete geometry generation under complex interactions and occlusions. Moreover, we introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains to obtain photo-realistic results. Extensive experiments demonstrate the effectiveness and efficiency of our approach in synthesizing photo-realistic free-view results under complex human-object interactions.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Neural Human Reconstructionmentioning

confidence: 99%

Section: Neural Human Reconstructionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

Jiang¹,

Jiang²,

Sun³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

SADNet: Generating immersive virtual reality avatars by real‐time monocular pose estimation

Jiang,

Xiong,

Wang

et al. 2024

Computer Animation & Virtual

View full text Add to dashboard Cite

SummaryGenerating immersive virtual reality avatars is a challenging task in VR/AR applications, which maps physical human body poses to avatars in virtual scenes for an immersive user experience. However, most existing work is time‐consuming and limited by datasets, which does not satisfy immersive and real‐time requirements of VR systems. In this paper, we aim to generate 3D real‐time virtual reality avatars based on a monocular camera to solve these problems. Specifically, we first design a self‐attention distillation network (SADNet) for effective human pose estimation, which is guided by a pre‐trained teacher. Secondly, we propose a lightweight pose mapping method for human avatars that utilizes the camera model to map 2D poses to 3D avatar keypoints, generating real‐time human avatars with pose consistency. Finally, we integrate our framework into a VR system, displaying generated 3D pose‐driven avatars on Helmet‐Mounted Display devices for an immersive user experience. We evaluate SADNet on two publicly available datasets. Experimental results show that SADNet achieves a state‐of‐the‐art trade‐off between speed and accuracy. In addition, we conducted a user experience study on the performance and immersion of virtual reality avatars. Results show that pose‐driven 3D human avatars generated by our method are smooth and attractive.

show abstract

AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture

Zheng

Zhang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

Cited by 195 publications

References 47 publications

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

NeuralHOFusion: Neural Volumetric Rendering under Human-object Interactions

SADNet: Generating immersive virtual reality avatars by real‐time monocular pose estimation

AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture

Contact Info

Product

Resources

About