Sequential Non-Rigid Structure from Motion Using Physical Priors

Agudo, Antonio; Moreno-Noguer, Francesc; Calvo, Begoña; Montiel, José María

doi:10.1109/tpami.2015.2469293

Cited by 71 publications

(82 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most approaches have required the use of additional priors using different optimization schemes to include temporal smoothness (Agudo et al 2012b;Bartoli et al 2008;Del Bue et al 2006;Torresani et al 2008;Vicente and Agapito 2012), smooth-time trajectories (Akhter et al 2011;Gotardo and Martinez 2011), inextensibility constraints (Vicente and Agapito 2012), rigid priors (Agudo et al 2016) and spatial smoothness (Garg et al 2013a;Torresani et al 2008). BA has become a popular optimization tool for refining an initial rigid solution (Tomasi and Kanade 1992) optimizing the pose, shape basis and coefficients while incorporating both motion and deformation priors (Bartoli et al 2008;Del Bue et al 2006).…”

Section: Related Workmentioning

confidence: 99%

Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

2016

Self Cite

View full text Add to dashboard Cite

This paper describes two sequential methods for recovering the camera pose together with the 3D shape of highly deformable surfaces from a monocular video. The non-rigid 3D shape is modeled as a linear combination of mode shapes with time-varying weights that define the shape at each frame and are estimated on-the-fly. The low-rank constraint is combined with standard smoothness priors to optimize the model parameters over a sliding window of image frames. We propose to obtain a physics-based shape basis using the initial frames on the video to code the timevarying shape along the sequence, reducing the problem from trilinear to bilinear. To this end, the 3D shape is discretized by means of a soup of elastic triangular finite elements where we apply a force balance equation. This equation is solved using modal analysis via a simple eigenvalue problem to obtain a shape basis that encodes the modes of deformation. Even though this strategy can be applied in a wide variety of scenarios, when the observations are denser, the solution can become prohibitive in terms of computational load. We avoid this limitation by proposing two efficient coarse-tofine approaches that allow us to easily deal with dense 3D surfaces. This results in a scalable solution that estimates a small number of parameters per frame and could potentially run in real time. We show results on both synthetic and real videos with ground truth 3D data, while robustly dealing with artifacts such as noise and missing data.

show abstract

Section: Related Workmentioning

confidence: 99%

Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

2016

Self Cite

View full text Add to dashboard Cite

show abstract

“…While sequential solutions exist for the rigid case (Newcome and Davison 2010;Lim et al 2011), sequential estimation of deformable objects based only on the measurements up to that moment remains a challenging and unsolved problem. There are just a few attempts along this direction (Agudo et al 2016(Agudo et al , 2014a(Agudo et al , 2012Paladini et al 2010;Tao et al 2013). Specifically, Paladini et al (2010) proposed a 3D-implicit low-rank model to encode the time-varying shape, estimating the remaining model parameters by BA over a temporal sliding window.…”

Section: Related Workmentioning

confidence: 99%

“…Based on a similar framework, Tao et al (2013) proposed an incremental principal component analysis to recursively update the low-rank model. Linear elasticity by means of finite element models was introduced into an extended Kalman filter to encode extensible deformations in real time (Agudo et al 2012), even computing the full camera trajectory (Agudo et al 2016). Very recently, (Agudo et al 2014a,b) presented the first approach to reconstruct both sparse and dense 3D shapes in a sequential manner, relying on a linear subspace of basis shapes computed by modal analysis.…”

Section: Related Workmentioning

confidence: 99%

“…Very recently, (Agudo et al 2014a,b) presented the first approach to reconstruct both sparse and dense 3D shapes in a sequential manner, relying on a linear subspace of basis shapes computed by modal analysis. However, despite being very promising, these methods are only valid to handle smoothly deforming objects, as is the case of (Paladini et al 2010;Tao et al 2013), and cannot be applied to articulated motion (Agudo et al 2012(Agudo et al , 2014b(Agudo et al , 2016). …”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

Agudo

Moreno-Noguer

2016

Int J Comput Vis

Self Cite

View full text Add to dashboard Cite

In this paper, we simultaneously estimate camera pose and non-rigid 3D shape from a monocular video, using a sequential solution that combines local and global representations. We model the object as an ensemble of particles, each ruled by the linear equation of the Newton's second law of motion. This dynamic model is incorporated into a bundle adjustment framework, in combination with simple regularization components that ensure temporal and spatial consistency. The resulting approach allows to sequentially estimate shape and camera poses, while progressively learning a global low-rank model of the shape that is fed back into the optimization scheme, introducing thus, global constraints. The overall combination of local (physical) and global (statistical) constraints yields a solution that is both efficient and robust to several artifacts such as noisy and missing data or sudden camera motions, without requiring any training data at all. Validation is done in a variety of real application domains, including articulated and non-rigid motion, both for continuous and discontinuous shapes. Our on-line methodology yields significantly more accurate reconstructions than competing sequential approaches, being even comparable to the more computationally demanding batch methods.

show abstract

“…The problem becomes even more challenging when input data is corrupted by artifacts such as noise or missing data. Over the past decade, a wide body of research has been proposed to tackle these complex situations [6,22,26,39]. At the core of most these methods, lies the assumption that objects do not arbitrarily change their shapes, and that their deformations can be ruled by low-rank models.…”

Section: Introductionmentioning

confidence: 99%

Global Model with Local Interpretation for Dynamic Shape Reconstruction

Agudo

Moreno-Noguer

2017

2017 IEEE Winter Conference on Applications of Computer Vision (WACV)

Self Cite

View full text Add to dashboard Cite

The most standard approach to resolve the inherent ambiguities of the non-rigid structure from motion problem is using low-rank models that approximate deforming shapes by a linear combination of rigid basis. These models are typically global, i.e., each shape basis contributes equally to all points of the surface. While this approach has been shown effective to represent smooth deformations, its performance degrades for surfaces composed of various regions, each following a different deformation rule. Piecewise methods attempt to capture this type of behavior by locally modeling surface patches, although they subsequently require enforcing global constraints to assemble back the patches. In this paper we propose an approach that combines the best of global and local models: it locally considers low-rank models but, by construction, does not need to impose global constraints to guarantee local patch continuity. We achieve this by a simple expectation maximization strategy that besides learning global shape bases, it locally adapts their contribution to each specific surface region. Furthermore, as a side contribution, in order to split the surface into different local patches, we propose a novel physically-based mesh segmentation approach that obeys an energy criterion. The complete framework is evaluated in both synthetic and real datasets, and shows an improved performance to competing methods.

show abstract

Sequential Non-Rigid Structure from Motion Using Physical Priors

Cited by 71 publications

References 47 publications

Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

Modal Space: A Physics-Based Model for Sequential Estimation of Time-Varying Shape from Monocular Video

Combining Local-Physical and Global-Statistical Models for Sequential Deformable Shape from Motion

Global Model with Local Interpretation for Dynamic Shape Reconstruction

Contact Info

Product

Resources

About