Learning Spatially-Smooth Mappings in Non-Rigid Structure From Motion

Abstract

Non-rigid structure from motion (NRSFM) is a classical underconstrained problem in computer vision. A common approach to make NRSFM more tractable is to constrain 3D shape deformation to be smooth over time. This constraint has been used to compress the deformation model and reduce the number of unknowns that are estimated. However, temporal smoothness cannot be enforced when the data lacks temporal ordering and its benefits are less evident when objects undergo abrupt deformations. This paper proposes a new NRSFM method that addresses these problems by considering deformations as spatial variations in shape space and then enforcing spatial, rather than temporal, smoothness. This is done by modeling each 3D shape coefficient as a function of its input 2D shape. This mapping is learned in the feature space of a rotation invariant kernel, where spatial smoothness is intrinsically defined by the mapping function. As a result, our model represents shape variations compactly using custom-built coefficient bases learned from the input data, rather than a pre-specified set such as the Discrete Cosine Transform. The resulting kernel-based mapping is a by-product of the NRSFM solution and leads to another fundamental advantage of our approach: for a newly observed 2D shape, its 3D shape is recovered by simply evaluating the learned function.

Robust deformable shape reconstruction from monocular video with manifold forests

“…cluster), can be modelled in the same linear manifold embedded in R N , and therefore all corresponding reconstructed shapes (represented by that cluster) can be approximated by a linear combination of the same set of unknown but fixed basis shapes. Thus all the shapes in the cluster i can be represented as The parameters θ tl , B i l and R t are optimised simultaneously by minimising the following modified cost function, (16) where the additional constraint applied to the i th set of basis shapes is, Figure 5 shows the embedding using reduced number of training samples. 40 shapes are randomly selected from the cardboard dataset.…”

“…Gotardo and Martinez demonstrated the "kernel trick" which used for nonlinear dimensionality reduction [31] can also be applied to standard NRSfM problem [15]. Recently Hamsici et al [16] modelled the shape coefficients in a manifold feature space. This method has ability to recover shapes from a newly observed image.…”

“…Similarly because of different depths, similar 2D images may not represent similar 3D shapes. In comparison, [15] defines a nonlinear model while [16] models 3D shapes in a linear space; [15] uses point trajectory bases as input data for building a kernel function while [16] uses shapes directly from 2D images.…”

An Accurate Online Non-rigid Structure from Motion Algorithm

“…No matter rigid SFM or NRSFM, the solutions are mostly obtained via the off-line model in the past decade [3][4][5]. In some real situation, e.g.…”

3D Reconstruction of Human Motion from Monocular Image Sequences

“…[2], [3], [4]) work well as long as there is a camera rotation around the observed object. Due to ambiguity in camera placement and 3D shape deformation they fail in realistic scenes such as a fixed camera filming a person walking by as shown in Fig.…”