Global Model with Local Interpretation for Dynamic Shape Reconstruction

Abstract

The most standard approach to resolve the inherent ambiguities of the non-rigid structure from motion problem is using low-rank models that approximate deforming shapes by a linear combination of rigid basis. These models are typically global, i.e., each shape basis contributes equally to all points of the surface. While this approach has been shown effective to represent smooth deformations, its performance degrades for surfaces composed of various regions, each following a different deformation rule. Piecewise methods attempt to capture this type of behavior by locally modeling surface patches, although they subsequently require enforcing global constraints to assemble back the patches. In this paper we propose an approach that combines the best of global and local models: it locally considers low-rank models but, by construction, does not need to impose global constraints to guarantee local patch continuity. We achieve this by a simple expectation maximization strategy that besides learning global shape bases, it locally adapts their contribution to each specific surface region. Furthermore, as a side contribution, in order to split the surface into different local patches, we propose a novel physically-based mesh segmentation approach that obeys an energy criterion. The complete framework is evaluated in both synthetic and real datasets, and shows an improved performance to competing methods.

Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints

“…A and traj. B, with 28, 000 points per frame) [19], expressions (384 frames with 997 points per frame) [4], and Kinect t-shirt (313 frames with 77, 000 points) and paper (193 frames with 58, 000 points) sequences taken from [64]. In the case if 3D ground truth shapes are available, ground truth dense point tracks are obtained by a virtual orthographic camera.…”

2D-to-3D Facial Expression Transfer

“…For every dataset, we indicate by (V/F) the number of vertices and triangular faces, respectively. We consider the face synthetic sequences denoted as Seq3 [26] (28,887/57,552), and Ogre [27] (19,985/39,856); the mocap video denoted as Mocap [21] (2,494/4,339); and finally the real face videos denoted as Face [26] (28,332/56,516), Face1 [28] (196,446/391,642) and Face2 [28] (196,446/391,614). In addition, we also consider the blending shapes (5,792/10,221) provided by [28].…”

“…A priori, no information about the video is assumed and hence the observed face could potentially undergo non-rigid motions. In the last decade, this problem has been addressed by non-rigid structure from motion approaches [19], [20], [21], showing accurate results even with incomplete 2D point tracks. However, our approach only requires a 3D model or rest shape, rather than knowing a 3D shape per image frame, which can be easily computed by rigid structure from motion techniques [22].…”

Robust Spatio-Temporal Clustering and Reconstruction of Multiple Deformable Bodies

“…The most standard approach to address the inherent ambiguity of the NRSfM problem is by enforcing the underlying 3D shape to lie in a low-rank manifold. In order to estimate such low-rank model, factorization-based approaches have been typically used [6], [15], [26], [31], [34], [48], [56]. Other approaches impose the low-rank constraints by means of robust PCA-like formulations which seek to minimize the rank of a matrix representing the shape.…”