Jumping Manifolds: Geometry Aware Dense Non-Rigid Structure From Motion

Abstract

Given dense image feature correspondences of a nonrigidly moving object across multiple frames, this paper proposes an algorithm to estimate its 3D shape for each frame. To solve this problem accurately, the recent state-ofthe-art algorithm reduces this task to set of local linear subspace reconstruction and clustering problem using Grassmann manifold representation [34]. Unfortunately, their method missed on some of the critical issues associated with the modeling of surface deformations, for e.g., the dependence of a local surface deformation on its neighbors. Furthermore, their representation to group high dimensional data points inevitably introduce the drawbacks of categorizing samples on the high-dimensional Grassmann manifold [32,31]. Hence, to deal with such limitations with [34], we propose an algorithm that jointly exploits the benefit of high-dimensional Grassmann manifold to perform reconstruction, and its equivalent lower-dimensional representation to infer suitable clusters. To accomplish this, we project each Grassmannians onto a lower-dimensional Grassmann manifold which preserves and respects the deformation of the structure w.r.t its neighbors. These Grassmann points in the lower-dimension then act as a representative for the selection of high-dimensional Grassmann samples to perform each local reconstruction. In practice, our algorithm provides a geometrically efficient way to solve dense NRSfM by switching between manifolds based on its benefit and usage. Experimental results show that the proposed algorithm is very effective in handling noise with reconstruction accuracy as good as or better than the competing methods.

Neural Dense Non-Rigid Structure from Motion with Latent Space Constraints

“…e 3D for the synthetic faces are reported in Table 1. We compare our N-NRSfM to Metric Projections (MP) [43], Trajectory Basis (TB) approach [7], Variational Approach (VA) [19], Dense Spatio-Temporal Approach (DSTA) [15], Coherent Depth Fields (CDF) [23], Consolidating Monocular Dynamic Reconstruction (CMDR) [24,25], Grassmannian Manifold (GM) [37], Jumping Manifolds (JM) [36], Scalable Monocular Surface Reconstruction (SMSR) [8], Expectation-Maximisation Finite Element Method (EM-FEM) [1] and Probabilistic Point Trajectory Approach (PPTA) [6]. Our N-NRSfM comes close to the most accurate methods on traj.…”

Neural Architecture Search of SPD Manifold Networks

“…Evaluation on several benchmark datasets showed the clear superiority of the proposed NAS method over handcrafted SPD networks and Euclidean NAS algorithms. As a future work, it would be interesting to generalize our NAS algorithm to cope with other manifold valued data (e.g., Chakraborty et al, 2018;Kumar et al, 2018;Kumar, 2019;Kumar et al, 2020]) and manifold poolings (e.g., Engin et al, 2018;), which are generally valuable for visual recognition, structure from motion, medical imaging, radar imaging, forensics, appearance tracking to name a few.…”

Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo

“…In the past, many active and passive 3D reconstruction approaches or pipelines were proposed to solve 3D reconstruction of objects [78,100,24,129,75,56,55]. However, when it comes to the accuracy of recovered 3D shapes for its use in scientific and engineering purposes (metrology), methods that use only MVS or PS suffer [76,90,25,26].…”