Self-Calibrating Isometric Non-Rigid Structure-from-Motion

Abstract

Non-Rigid Structure-from-Motion (NRSfM) reconstructs a deformable 3D object from the correspondences established between monocular 2D images. Current NRSfM methods lack statistical robustness, which is the ability to cope with correspondence errors. This prevents one to use automatically established correspondences, which are prone to errors, thereby strongly limiting the scope of NRSfM. We propose a three-step automatic pipeline to solve NRSfM robustly by exploiting isometry. Step (i) computes the optical flow from correspondences, step (ii) reconstructs each 3D point's normal vector using multiple reference images and integrates them to form surfaces with the best reference and step (iii) rejects the 3D points that break isometry in their local neighborhood. Importantly, each step is designed to discard or flag erroneous correspondences. Our contributions include the robustification of optical flow by warp estimation, new fast analytic solutions to local normal reconstruction and their robustification, and a new scale-independent measure of 3D local isometric coherence. Experimental results show that our robust NRSfM method consistently outperforms existing methods on both synthetic and real datasets.

Total Estimation from RGB Video: On-line Camera Self-Calibration, Non-Rigid Shape and Motion

“…While these formulations normally use a perspective camera model, the internal parameters have to be known a priori, being a good calibration a key factor to achieve accurate solutions. Enforcing also inextensibility, some works [14], [33] have been also extended to include the focal length in the estimation.…”

“…Unfortunately, these approaches rely on orthographic camera models, or they are not capable of solving feature tracking and outliers detection in a single process. More recently, in [8] was proposed a sequential solution to recover both inelastic and [19], [25], [38] [2], [32] [8] [14], [33] Ours elastic materials while tracks the feature points. In addition, this approach includes a full projective camera model where the calibration is pre-computed after video capture.…”

Unsupervised Monocular Depth Reconstruction of Non-Rigid Scenes

“…Other related geometric priors have also been exploited in the literature [46,60,1]. When objects undergo severe deformations, relying on a geometric prior for the objects and avoiding any explicit camera motion estimation have demonstrated state-of-the art results for non-rigid object reconstruction [55,52,58,22,47]. The benefit of not estimating the camera motion explicitly can be understood quite intuitively, as the estimated rigid camera motion is merely a motion with respect to the scene.…”

A Closed-Form, Pairwise Solution to Local Non-Rigid Structure-from-Motion

“…Reconstructing the 3D shape of deformable objects from monocular image sequences is known as Non-Rigid Structure-from-Motion (NRSfM) and has applications in domains ranging from entertainment [35] to medicine [26]. Early methods relied on lowrank representations of the surfaces [4], [7], [10], [12], [17], [23], [25], [28], [49], while more recent ones exploit local surface properties to derive constraints and can handle larger deformations [8], [9], [20], [47], [50], [51].…”