HDM-Net: Monocular Non-rigid 3D Reconstruction with Learned Deformation Model

Abstract

Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive. Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call Hybrid Deformation Model Network (HDM-Net). In our approach, a deformation model is learned by a deep neural network, with a combination of domainspecific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, shading and contours. We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under new illumination conditions. Experiments with noisy data and a comparison with other methods demonstrate the robustness and accuracy of the proposed approach and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality.

IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction

“…At the same time, IsMo-GAN is trained in the supervised manner. The training dataset contains a sequence of 3D states along with the corresponding 2D images [17]. Thus, our framework bears a remote analogy with SfT, as IsMo-GAN is trained for a deformation model with a pre-defined surface at rest (or multiple surfaces at rest, in the extended version).…”

DispVoxNets: Non-Rigid Point Set Alignment with Supervised Learning Proxies

“…of the inputs, which poses challenges in the network architecture design. Another reason is that sufficiently comprehensive collections of deformable shapes with large deformations, suitable for the training have just recently become available [5,2,17,36].…”

“…Ours NR-ICP [9] CPD [41] GMR [31] thin plate [17] ref. Table 5: Registration errors for the case with missing parts.…”

“…In Fig. II, we show selected registrations by our approach and other tested methods (NR-ICP [9], CPD/CPD with FGT [41], and GMR [31]) on the tested datasets (thin plate [17], FLAME [36], DFAUST [5] and cloth [2]).…”

“…Since our technique learns class-specific priors implicitly during training, it is successful in registering samples with large displacements and articulations. Figure III: RMSE (e) graphs for additional experiments with uniform noise on thin plate [17], FLAME [36], DFAUST [5] and cloth [2] datasets. p% is the ratio between the number of added points and the number of points in the sample.…”