Human Performance Capture from Monocular Video in the Wild

Abstract

Capturing the dynamically deforming 3D shape of clothed human is essential for numerous applications, including VR/AR, autonomous driving, and human-computer interaction. Existing methods either require a highly specialized capturing setup, such as expensive multi-view imaging systems, or they lack robustness to challenging body poses. In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input. We first build a 3D template human model of the subject based on a learned regression model. We then track this template model's deformation under challenging body articulations based on 2D image observations. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW. Moreover, we demonstrate its efficacy in robustness and generalizability on videos from iPER datasets.

Efficient Meshy Neural Fields for Animatable Human Avatars

“…Need foreground mask to enable the mesh optimization, akin to shape-from-silhouette. One future direction might be equipping our method with the ability to separate foreground and background automatically [31,24]. It is also promising to model the background simultaneously during foreground subject optimization [31,24], which eliminates the requirement of foreground mask processing.…”

PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence

“…These works typically leverage parametric models for minimally clothed human bodies [9,17,18,29,46,55] (e.g. SMPL [33]) and use a displacement layer on top of the minimally clothed body to model clothing [3,4,23,35,39,61]. Recently, DSFN [11] proposes to embed MLPs into the canonical space of SMPL to model pose-dependent deformations.…”

PINA: Learning a Personalized Implicit Neural Avatar from a Single RGB-D Video Sequence

“…These works typically leverage parametric models for minimally clothed human bodies [7,15,16,27,44,52] (e.g. SMPL [31]) and use a displacement layer on top of the minimally clothed body to model clothing [3,4,21,33,37,57]. Recently, DSFN [9] proposes to embed MLPs into the canonical space of SMPL to model pose-dependent deformations.…”