Segmentation and 3D Reconstruction of NON-RIGID Shape from RGB Video

Abstract

In this paper we propose a unsupervised and unified approach to simultaneously recover time-varying 3D shape, camera motion, and temporal clustering into deformations, all of them, from partial 2D point tracks in a RGB video and without assuming any pre-trained model. As the data are drawn from a sequentially ordered images, we fully exploit this information to constrain all model parameters we estimate. We present an energy-based formulation that is efficiently solved and allows to estimate all model parameters in the same loop via augmented Lagrange multipliers in polynomial time, enforcing similarities between images at any level. Validation is done in a wide variety of human video sequences, including articulated and continuous motion, and for dense and missing tracks. Our approach is shown to outperform state-of-the-art solutions in terms of 3D reconstruction and clustering.

Piecewise Bézier Space: Recovering 3D Dynamic Motion From Video

“…Finally, we also validate our approach on dense data by running two videos with 20,561 and 68,295 2D point trajectories taken from [17], where a back and a heart are deforming and moving, respectively. In the same figure is represented our joint solution in these videos, obtaining qualitatively accurate and physically possible solutions in comparison to [23]. In all cases, we display the same information.…”

“…We now present our experimental evaluation on several human motion videos, including articulated and continuous deformation, several body configurations and scenarios with missing or dense entries. For quantitative comparison, we apply our approach on the articulated human motion dataset introduced in [5], which includes five types of activities, and nine competing methods are considered: EM-PPCA [4], MP [2], PTA [5], CSF [8], KSTA [7], BMM [16], PPTA [27], URS [18], and TRUS [23]; under two scenarios: noise-free and noisy 2D point trajectories as it was done in [27]. As in the literature [8,16,27], we provide the normalized mean 3D error e S , and the mean rotation error e R .…”

“…In contrast to other approaches, this is (7) .330 .357 (12) .006 .025 (13) .006 .022 (6) .006 .020 (12) .007 .027 (12) .006 .011 (30) .006 .009 0.8 (2) .006 .009 0.6 (2) .005 .009 0.6(2) Stretch .749 .458 (7) .832 .900 (8) .055 .109 (12) .049 .071 (8) .049 .064 (11) .068 .103 (11) .058 .084 (11) .058 . (7) .329 .517 (12) .043 .045 (13) .043 .044 (6) .043 .042 (12) .044 .056 (12) .042 .038 (30) [5], CSF [8], KSTA [7], BMM [16], PPTA [27], URS [18], and TRUS [23]; and for our approach, considering both noise-free and noisy observations. The table also indicates in parentheses the rank K of the linear subspace that produced the lowest e S .…”

“…Thanks to those representations, some segmentations into motion activities or spatial bodies were included in the estimation. In order to improve robustness, on top of those shape models additional spatial [3] and temporal [4,22,23] smoothness priors have also been considered. In this paper, we depart from previous work in that our approach exploits a union of piecewise Bézier subspaces with enforcing continuities to model 3D point trajectories that are observed in a RGB video.…”

Spline Human Motion Recovery

“…The problem of simultaneously recovering the 3D reconstruction of a dynamic object and the camera motion from incomplete 2D point tracks in a video is coined in the literature as non-rigid structure from motion [1,2,3]. Despite being a very challenging problem with many real-world applications in several domains, solving it without 3D supervision is an ill-posed problem that requires exploring the art of priors to be tractable.…”