IsMo-GAN: Adversarial Learning for Monocular Non-Rigid 3D Reconstruction

Abstract

The majority of the existing methods for non-rigid 3D surface regression from monocular 2D images require an object template or point tracks over multiple frames as an input, and are still far from real-time processing rates. In this work, we present the Isometry-Aware Monocular Generative Adversarial Network (IsMo-GAN) -an approach for direct 3D reconstruction from a single image, trained for the deformation model in an adversarial manner on a light-weight synthetic dataset. IsMo-GAN reconstructs surfaces from real images under varying illumination, camera poses, textures and shading at over 250 Hz. In multiple experiments, it consistently outperforms several approaches in the reconstruction accuracy, runtime, generalisation to unknown surfaces and robustness to occlusions. In comparison to the state-of-the-art, we reduce the reconstruction error by 10-30% including the textureless case and our surfaces evince fewer artefacts qualitatively.

Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints

“…Another possibility to perform monocular non-rigid 3D reconstruction is to use learning-based approaches. Recently, many works have been presented for rigid [13,18,30,40,66] and non-rigid [27,47,54,62] shape reconstruction. These methods exploited a large and annotated dataset to learn the solution space, limiting their applicability to the type of shapes that are observed in the dataset.…”

“…Dense NRSfM has achieved remarkable progress during the last several years [1,8,19,37,51]. While the accuracy of dense NRSfM has been recently only marginally improved, learning-based direct methods for monocular rigid and non-rigid 3D reconstruction have become an active research area in computer vision [13,33,47,54,66].…”

Texture-Generic Deep Shape-From-Template

“…In recent years, the idea of using learning methods to solve SfT has been explored. These methods learn the mapping from the input image to the object 3D deformation [19][20][21][22]. They are potentially able to solve SfT in wide-baseline conditions and without the need to run optimisation at runtime.…”

“…First, some of them only work with a specific template [22], requiring fine tuning for the specific template shape and texture map. Other works propose to solve SfT for a variety of texture maps [19][20][21], learning invariance to these. However, their results are far from satisfactory, still requiring fine tuning of the network to a specific template.…”

“…Second, in terms of template geometry, most existing methods tackle the thin-shell case [19][20][21], with the exception of the template-specific method [22]. Third, in terms of template complexity, most methods require a mesh with a reduced number of vertices (namely, 73×73 in [20,21] and 10 × 10 in [19]), which is a strong limitation to work with complex templates or deformations. Fourth, they work for a specific camera configuration, not being able to adapt to dynamical camera parameters at run-time.…”