Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

Abstract

This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The non-rigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the timevarying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-tofine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.

DUST: Dual Union of Spatio-Temporal Subspaces for Monocular Multiple Object 3D Reconstruction

“…These type of methods either assume the data lies on a single low dimensional space [16,18,20] or in a union of temporal subspaces [40]. On top of these shape models, additional spatial [24] or temporal [1,8,26] smoothness constraints have also been considered. Low-rank models have been extended to the temporal domain, by fitting point trajectories to a series of predefined basis [6,32,38], to shape-and-temporal composite domains [21,22,35], and to the space of forces that induce the deformations [3].…”

Global Model with Local Interpretation for Dynamic Shape Reconstruction

“…are computed following [3]. We can then solve equilibrium equations and obtain the undamped free vibration response of the 3D object caused by a disturbance with respect to the shape at rest s 0 based on the following generalized eigenvalue problem:…”

“…Particularly, we consider: EM-LDS [39], and EM-PND [26] for shape space; PTA [7] for trajectory space; the Column Space Fitting (CSF2) [23] and the Kernel Shape Trajectory Approach (KSTA) [22] for shapetrajectory methods 3 . The parameters of these methods were set as suggested in the original papers.…”

Recovering Pose and 3D Deformable Shape from Multi-instance Image Ensembles

“…The prior most widely used in NRSfM consists in constraining the shape to lie on a global lowrank shape subspace, that can be computed over a set of training data [25], applying modal [26,27] or spectral [28] analysis over a rest shape, or estimating it on-the-fly [10,12,29]. Most approaches build upon the well-known closedform factorization technique used for rigid reconstruction [30], enforcing camera orthonormality constraints.…”