DUST: Dual Union of Spatio-Temporal Subspaces for Monocular Multiple Object 3D Reconstruction

Abstract

We present an approach to reconstruct the 3D shape of multiple deforming objects from incomplete 2D trajectories acquired by a single camera. Additionally, we simultaneously provide spatial segmentation (i.e., we identify each of the objects in every frame) and temporal clustering (i.e., we split the sequence into primitive actions). This advances existing work, which only tackled the problem for one single object and non-occluded tracks. In order to handle several objects at a time from partial observations, we model point trajectories as a union of spatial and temporal subspaces, and optimize the parameters of both modalities, the nonobserved point tracks and the 3D shape via augmented Lagrange multipliers. The algorithm is fully unsupervised and results in a formulation which does not need initialization. We thoroughly validate the method on challenging scenarios with several human subjects performing different activities which involve complex motions and close interaction. We show our approach achieves state-of-the-art 3D reconstruction results, while it also provides spatial and temporal segmentation.

Deep Non-Rigid Structure From Motion

“…Loss function: The loss function must measure the reprojection error between input 2D points W and reprojected 2D points SM while simultaneously encouraging orthonormality of the estimated camera M. One solution is to use 1 The filter dimension is height×width×# of input channel×# of output channel. The feature dimension is height×width×# of channel.…”

“…They show that clustering frames from their 2D annotations is less effective and therefore propose a novel algorithm to reconstruct 3D shapes and an estimate of the 3D-based frame affinity matrix simultaneously. The idea of union-of-subspaces was later extended to spatial-temporal domain [19] and applied to rigid object category reconstruction [7].Though the unionof-subspaces model is capable of handling complex object deformation, its holistic estimation of the entire affinity matrix -the number of frame by the number of frame matrix -impedes its scalability to large-scale problems e.g. more than tens of thousand frames.…”

Image Collection Pop-up: 3D Reconstruction and Clustering of Rigid and Non-rigid Categories

“…More recently, the low-rank constraint has been imposed by means of PCA-like formulations in which the rank of the shape matrix is optimized. These type of methods either assume the data lies in a single low dimensional shape space [16,19,21], or in a union of temporal [41] or spatio-temporal subspaces [6]. Low-rank models were also extended to the temporal domain, by exploiting pre-defined trajectory basis [7,35], the combination of shape-trajectory domains [22,23], and the force space that induces the deformations [3].…”

Scalable Dense Non-rigid Structure-from-Motion: A Grassmannian Perspective

“…b) Why the previously proposed spatio-temporal methods are unable to handle dense NRSfM? The formulation proposed by Kumar et al [23] and its adaptation [3] is inspired from SSC [16], and LRR [30]. As a result, the complexity of their formulations grows exponentially in the order of the number of data points.…”

“…This makes it difficult to solve dense NRSfM using their formulations. Moreover, these methods [23,41,3] use an assumption that non-rigid shape should lie on a low-dimensional linear or affine subspace globally. In reality, such an assumption does not hold for all kinds of non-linear deformations [39,34].…”