DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor

Abstract

We propose DoubleFusion, a new real-time system that combines volumetric dynamic reconstruction with datadriven template fitting to simultaneously reconstruct detailed geometry, non-rigid motion and the inner human body shape from a single depth camera. One of the key contributions of this method is a double layer representation consisting of a complete parametric body shape inside, and a gradually fused outer surface layer. A pre-defined node graph on the body surface parameterizes the nonrigid deformations near the body, and a free-form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance. Moreover, the inner body shape is optimized online and forced to fit inside the outer surface layer. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. In particular, experiments show improved fast motion tracking and loop closure performance on more challenging scenarios.

Target Image Mask Correction Based on Skeleton Divergence

“…Fourth, considering the huge computational complexity of dealing with dense point clouds, the acquisition of the matching degree of the point cloud regularization constraint is based on the sparse point cloud. In the same way as the method in [17], with the help of incremental and hierarchical clustering, and iterative simplification, these point clouds are simplified in advance so that they are concentrated in the high-curvature region, a surface deformation model is established [18], and then the system accuracy is guaranteed while reducing the amount of calculation in the process. Finally, with the regional constraints of the global skeleton and boundary, combined with the morphological features and structural similarity, the hole filling of the merged discrete pixel regions is performed to improve the accuracy of the binary mask correction and efficiency.…”

DeepHuman: 3D Human Reconstruction From a Single Image

“…Many of the following approaches have tried to improve the robustness by adding color features [23], shading constraints [19] and articulated prior [75] and dealing with topology changes [56,57]. The recently appeared DoubleFusion [76] method introduced a human shape prior into the fusion pipeline and achieved state-of-the-art real-time efficiency, robustness, and loop closure performance for efficient human model reconstruction even in cases of fast motions. There are also offline methods for global registration of multiple RGBD images to obtain a full-body model [35].…”

“…To fill in this gap, we present the THuman dataset. We leverage the state-of-theart DoubleFusion [76] technique for real-time human mesh reconstruction and propose a capture pipeline for fast and efficient capture of outer geometry of human bodies wearing casual clothes with medium-level surface detail and texture. Based on this pipeline, we perform capture and reconstruction of the THuman dataset, which contains about 7000 human meshes with approximately 230 kinds of clothes under randomly sampled poses.…”

Learning and Tracking the 3D Body Shape of Freely Moving Infants from RGB-D sequences

“…Yu et al [48] introduce an approach for real-time reconstruction of non-rigid surface motion from RGB-D data using skeleton information to regularize the shape deformations. They extend this approach by combining a parametric body model to represent the inner body shape with a freely deformable outer surface layer capturing surface details [49].…”