Robust Spatio-Temporal Clustering and Reconstruction of Multiple Deformable Bodies

Abstract

In this paper we present an approach to reconstruct the 3D shape of multiple deforming objects from a collection of sparse, noisy and possibly incomplete 2D point tracks acquired by a single monocular camera. Additionally, the proposed solution estimates the camera motion and reasons about the spatial segmentation (i.e., identifies each of the deforming objects in every frame) and temporal clustering (i.e., splits the sequence into motion primitive actions). This advances competing work, which mainly tackled the problem for one single object and non-occluded tracks. In order to handle several objects at a time from partial observations, we model point trajectories as a union of spatial and temporal subspaces, and optimize the parameters of both modalities, the non-observed point tracks, the camera motion, and the time-varying 3D shape via augmented Lagrange multipliers. The algorithm is fully unsupervised and does not require any training data at all. We thoroughly validate the method on challenging scenarios with several human subjects performing different activities which involve complex motions and close interaction. We show our approach achieves state-of-the-art 3D reconstruction results, while it also provides spatial and temporal segmentation.

Notice of Retraction: An Improved High-Density Sub Trajectory Clustering Algorithm

“…Yu et al [29] proposed an enhanced trajectory model based on multi-feature trajectory similarity measure to better apply for traffic monitoring and traffic congestion prediction. Zhu et al [30] created a recurrent convolutional neural network for modeling the complex nonlinear relationship among features, which could be used to predict road traffic conditions [31]. Wang et al [32] obtained all the taxi trajectories crossing the same source-destination pairs, and then, the regular trajectories and anomalous trajectories were distinguished by applying adaptive hierarchical clustering based on an optimal number of clusters.…”

Low-Rank and Sparse Recovery of Human Gait Data

“…Most data recovery approaches in the literature are investigated for image repairing [46][47][48][49][50] and action recognition such as jumping [53,54,57]. In this study, we focused on the efficient reconstruction of the missing gait data for various fields of application such as clinical applications (medical rehabilitation, prosthetics, etc.)…”

“…Some studies successfully adopted the low-rank and sparse representation for various applications, such as image repairing [46][47][48][49][50] and action recognition, such as walking, jumping, etc. [51][52][53][54][55][56][57]. Compared to previous works utilizing standard techniques, their reconstruction results showed that a combination of low rank and sparse approach yields significant improvements.…”

Segmentation and 3D Reconstruction of NON-RIGID Shape from RGB Video

“…In order to address the problem, the most standard approach is to enforce a low-rank constraint over P, since the time-varying configurations in X can lie in a linear subspace of rank 3K [21]. However, it was observed that another interpretation can be considered, removing the use of unnecessary degrees of freedom [15,16,22]. To achieve that, we can define the matrixX that rearranges the entries of X into a new 3N × F matrix, where a K-rank constraint could be imposed.…”