Monocular Total Capture: Posing Face, Body, and Hands in the Wild

Abstract

Figure 1: We present the first method to simultaneously capture the 3D total body motion of a target person from a monocular view input. For each example, (left) input image and (right) 3D total body motion capture results overlaid on the input. AbstractWe present the first method to capture the 3D total motion of a target person from a monocular view input. Given an image or a monocular video, our method reconstructs the motion from body, face, and fingers represented by a 3D deformable mesh model. We use an efficient representation called 3D Part Orientation Fields (POFs), to encode the 3D orientations of all body parts in the common 2D image space. POFs are predicted by a Fully Convolutional Network (FCN), along with the joint confidence maps. To train our network, we collect a new 3D human motion dataset capturing diverse total body motion of 40 subjects in a multiview system. We leverage a 3D deformable human model to reconstruct total body pose from the CNN outputs by exploiting the pose and shape prior in the model. We also present a texture-based tracking method to obtain temporally coherent motion capture output. We perform thorough quantitative evaluations including comparison with the existing body-specific and hand-specific methods, and performance analysis on camera viewpoint and human pose changes. Finally, we demonstrate the results of our total body motion capture on various challenging in-the-wild videos. Our code and newly collected human motion dataset will be publicly shared.

Learnable Triangulation of Human Pose

“…We conduct experiments on two available large multiview datasets with available ground-truth 3D pose annotations: Human3.6M [3] and CMU Panoptic [5,20,15].…”

“…CMU Panoptic dataset. The CMU panoptic is a new multi-camera dataset maintained by the Carnegie Mellon University [5,20,15]. The dataset provides 30 Hz Full-HD videostreams of 40 subjects from up to 31 synchronized cameras.…”

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

“…Single-View 3D Human Digitization Single-view 3D human reconstruction is an ill-posed problem due to the fundamental depth ambiguity along camera rays. To overcome such ambiguity, parametric 3D models [5,27,18,33] are often used to restrict estimation to a small set of model parameters, constraining the solution space to a specifically chosen parametric body model [7,22,20,46,33,47]. However, the expressiveness of the resulting models is limited by using a single template mesh as well as by the data on which the model is built (often comprised mainly of minimally clothed people).…”

TexturePose: Supervising Human Mesh Estimation With Texture Consistency

“…Typically, as we ascend the pyramid of human understanding, we target more and more challenging tasks. As expected, the emergence of sophisticated parametric models of the human body, like SCAPE [6], SMPL(-X) [25,32,40], and Adam [17,51], has really paved the way for full 3D pose and shape estimation from image data. And while this step has been well explored for video or multi-view data [13,17], the ultimate goal is to reach the same level of analysis from a single image.…”