Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints

Abstract

We introduce the first dense neural non-rigid structure from motion (N-NRSfM) approach, which can be trained end-to-end in an unsupervised manner from 2D point tracks. Compared to the competing methods, our combination of loss functions is fully-differentiable and can be readily integrated into deep-learning systems. We formulate the deformation model by an auto-decoder and impose subspace constraints on the recovered latent space function in a frequency domain. Thanks to the state recurrence cue, we classify the reconstructed non-rigid surfaces based on their similarity and recover the period of the input sequence. Our N-NRSfM approach achieves competitive accuracy on widely-used benchmark sequences and high visual quality on various real videos. Apart from being a standalone technique, our method enables multiple applications including shape compression, completion and interpolation, among others. Combined with an encoder trained directly on 2D images, we perform scenario-specific monocular 3D shape reconstruction at interactive frame rates. To facilitate the reproducibility of the results and boost the new research direction, we open-source our code and provide trained models for research purposes 1 .

Piecewise Bézier Space: Recovering 3D Dynamic Motion From Video

“…2, the error reduction is consistent as K increases, doing the error always remains within reasonable bounds. In contrast to other approaches, this is (7) .330 .357 (12) .006 .025 (13) .006 .022 (6) .006 .020 (12) .007 .027 (12) .006 .011 (30) .006 .009 0.8 (2) .006 .009 0.6 (2) .005 .009 0.6(2) Stretch .749 .458 (7) .832 .900 (8) .055 .109 (12) .049 .071 (8) .049 .064 (11) .068 .103 (11) .058 .084 (11) .058 . (7) .329 .517 (12) .043 .045 (13) .043 .044 (6) .043 .042 (12) .044 .056 (12) .042 .038 (30) [5], CSF [8], KSTA [7], BMM [16], PPTA [27], URS [18], and TRUS [23]; and for our approach, considering both noise-free and noisy observations.…”

“…The NRSfM problem consists in factoring the measurement matrix W into the motion G and shape S factors, i.e., inferring camera pose and non-rigid 3D reconstruction from 2D point trajectories in a monocular video. Trajectory-based models [5,13,26,27] approximate the position of every point coordinate over time by a linear combination of K low-frequency basis vectors. This representation is global and cannot be adapted properly for those scenarios with piece-varying motion.…”

“…Maybe, the most popular priors are based on low-rank subspaces constraining the solution space of either the entire shape [1,2,3,4], the 3D point trajectories [5,6], shape-trajectory combinations [7,8,9,10] or the force patterns that induce the deformations [11]. Similar shape [12] and trajectory [13] subspaces have also been exploited in a deep learning context, where large amounts of training data are needed to learn the model. While shape and force bases need to be estimated from data, trajectory ones can be predefined a priori and are available for a wide range of deformations.…”

“…While shape and force bases need to be estimated from data, trajectory ones can be predefined a priori and are available for a wide range of deformations. Although several representations could be used, the discrete cosine transform is usually chosen to model the basis [5,6,13]. However, this representation is global and cannot be adapted properly for those scenarios with piecevarying motion.…”

Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes

“…Although a recent unsupervised method [39] also reconstructs objects translating on a ground plane, it has limitations in terms of modelling highly non-rigid scenes. Other unsupervised methods for non-rigid reconstruction exploit object-specific priors [59,77,80], or reconstruct sparse [49] or dense [64,66] points of a single non-rigid object. Such methods, however, do not have the same applicability or motivation as that of single view scene depth estimation.…”