Markerless Multiview Motion Capture with 3D Shape Model Adaptation

Abstract

In this paper, we address simultaneous markerless motion and shape capture from 3D input meshes of partial views onto a moving subject. We exploit a computer graphics model based on kinematic skinning as template tracking model. This template model consists of vertices, joints and skinning weights learned a priori from registered full‐body scans, representing true human shape and kinematics‐based shape deformations. Two data‐driven priors are used together with a set of constraints and cues for setting up sufficient correspondences. A Gaussian mixture model‐based pose prior of successive joint configurations is learned to soft‐constrain the attainable pose space to plausible human poses. To make the shape adaptation robust to outliers and non‐visible surface regions and to guide the shape adaptation towards realistically appearing human shapes, we use a mesh‐Laplacian‐based shape prior. Both priors are learned/extracted from the training set of the template model learning phase. The output is a model adapted to the captured subject with respect to shape and kinematic skeleton as well as the animation parameters to resemble the observed movements. With example applications, we demonstrate the benefit of such footage. Experimental evaluations on publicly available datasets show the achieved natural appearance and accuracy.

Split Rendering for Mixed Reality: Interactive Volumetric Video in Action

“…This provides local temporal stability while preserving the captured geometry and mesh silhouette. In order to make the volumetric data animatable, we fit a parametric human model [Fechteler et al 2016;Robertini et al 2016] to the captured data [Fechteler et al 2019]. Thereby, we enrich the captured data with semantic pose and animation data taken from the model, used to drive the animation of the captured VV data itself.…”

SparseFusion: Dynamic Human Avatar Modeling from Sparse RGBD Images

“…However, they did not take the surface deformation caused by cloth into account but assumed that the captured human subject is almost naked. In paper [16], [1], a kinematic skinning model was used for human pose and shape reconstruction from the 3D point cloud acquired by multi-view stereo methods. Alldieck et al [5], [6] took a monocular video sequence as input and exploited the SMPL model for coarse shape and pose estimation, together with the human silhouettes and image shading information for more detailed reconstruction.…”

Progressive Non-rigid Registration of Temporal Mesh Sequences

“…This normalbased heuristic removes many possible mismatches of close surfaces pointing in opposite directions. In registrations of humans and other articulated objects, the heuristic helps to avoid mismatches between body parts held closely together, such as an arm next to the chest or individual fingers [Fechteler et al 2019]. Figure 6: Histogram of the Euclidean distances between vertices of the registered mesh and the triangle surface of the target mesh for frame at t = 50 in the Josh sequence.…”

“…Human motion can be captured from the scene by fitting human models [Bogo et al 2017] [Fechteler et al 2019] onto meshes of the human body. Recent contributions work with outdoor sequences [Robertini et al 2016] and demonstrate real-time processing [Habermann et al 2019].…”