PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction

Abstract

Modeling 3D humans accurately and robustly from a single image is very challenging, and the key for such an ill-posed problem is the 3D representation of the human models. To overcome the limitations of regular 3D representations, we propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. In our PaMIR-based reconstruction framework, a novel deep neural network is proposed to regularize the free-form deep implicit function using the semantic features of the parametric model, which improves the generalization ability under the scenarios of challenging poses and various clothing topologies. Moreover, a novel depth-ambiguity-aware training loss is further integrated to resolve depth ambiguities and enable successful surface detail reconstruction with imperfect body reference. Finally, we propose a body reference optimization method to improve the parametric model estimation accuracy and to enhance the consistency between the parametric model and the implicit function. With the PaMIR representation, our framework can be easily extended to multi-image input scenarios without the need of multi-camera calibration and pose synchronization. Experimental results demonstrate that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.

Human Performance Capture from Monocular Video in the Wild

“…[23,27,38,25,15,43,26,47,31] learn to infer body pose and shape from a single image, but only consider minimally clothed human. Various methods [48,60,6,42,41,18,21,59,28,36,13] have recently been proposed to reconstruct human in clothing. BodyNet [48] and DeepHuman [60] output human shape in the form of occupancy voxel grids.…”

“…Such representation has difficulties to capture fine details due to the high memory footprint. Neural implicit functions have been introduced to replace an explicit voxel grid and have enabled high-fidelity reconstructions from single images [42,41,18,21,59,28]. A major limitation of these methods is the lack of generalization to unseen poses in the wild.…”

Human Motion Tracking with Less Constraint of Initial Posture from a Single RGB-D Sensor

“…Wang et al [ 29 ] introduced an adversarial learning framework based on normal maps, which not only improves the front view depth de-noising performance but also infers back view depth images with impressive geometric details. Onizuka et al [ 26 ] combined a CNN (convolutional neural networks) and PCN (corresponding part connection network) to learn a distribution of the TSDF in the tetrahedral volume from a single image. Huang et al [ 27 ] used parametric 3D human body estimation to construct the semantic space and semantic deformation field, which allows the 2D/3D human body to be converted into a canonical space to reduce geometric blur caused by occlusion in pose changes.…”

“…At present, the method that uses a single RGB image as the input is the mainstream, and the ambiguity of the scale of RGB images is an unavoidable limitation. Moreover, using only RGB images to restore the geometric details of the model does not seem to be a reliable method [ 21 , 22 , 23 , 24 , 26 , 27 , 28 ].…”

“…These data-driven methods already encode prior information about the human body, such as posture and body shape. However, the methods based on only RGB input [ 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 ] did not achieve a reliable body shape estimation due to the ambiguity of scale. For another reason, the methods that employed depth input [ 30 , 31 ] have also made some progress, but the recovery of details is unsatisfactory.…”