Self-Expressive Dictionary Learning for Dynamic 3D Reconstruction

Abstract

We target the problem of sparse 3D reconstruction of dynamic objects observed by multiple unsynchronized video cameras with unknown temporal overlap. To this end, we develop a framework to recover the unknown structure without sequencing information across video sequences. Our proposed compressed sensing framework poses the estimation of 3D structure as the problem of dictionary learning, where the dictionary is defined as an aggregation of the temporally varying 3D structures. Given the smooth motion of dynamic objects, we observe any element in the dictionary can be well approximated by a sparse linear combination of other elements in the same dictionary (i.e., self-expression). Our formulation optimizes a biconvex cost function that leverages a compressed sensing formulation and enforces both structural dependency coherence across video streams, as well as motion smoothness across estimates from common video sources. We further analyze the reconstructability of our approach under different capture scenarios, and its comparison and relation to existing methods. Experimental results on large amounts of synthetic data as well as real imagery demonstrate the effectiveness of our approach.

Discrete Laplace Operator Estimation for Dynamic 3D Reconstruction

“…Along these lines, Avidan and Shashua [6] estimate dynamic geometry from 2D observations of points constrained to linear and conical motions. However, under the assumption of dense temporal motion sampling, the concept of motion smoothness has been successfully exploited [25,26,45,46,35,42,43,36,30,31]. Park et al [25] triangulate 3D point trajectories by the linear combination of Direct Cosine Transform trajectory bases with the constraint of a reprojection system.…”

“…Valmadre et al [35] reduce the number of trajectory bases by setting a gain threshold depending on the basis null-space and propose a method using a highpass filter to mitigate low reconstructability for scenarios having no missing 2D observations. Zheng et al [43,42] propose a dictionary learning method to estimate the 3D shape with partial sequencing info, assuming 3D geometry estimates may be approximated by local barycentric interpolation (i.e. self-expressive motion prior) and developed a bi-convex framework for jointly estimating 3D geometry and barycentric weights.…”

“…Enforcing Observation Ray Constrains. We penalize the distance of a 3D point X np to its known viewing ray using d np = ||(X np − C n ) × r np || 2 , where r np is a unit vector parallel to the viewing ray R n K −1 n [x np 1] and camera pose parameters are given by M n = [R n | − R n C n ] [43]. The functional form of O from Eq.…”

“…Accuracy is quantified by mean 3D reconstruction error. Our method discrete Laplace operator estimation (DLOE) is compared against self-expressive dictionary learning (SEDL) [43], trajectory basis (TB) [26], high-pass filter (HPF) [35] and the pseudo-triangulation approach in Sec. 6.…”

Learning Structured Twin-Incoherent Twin-Projective Latent Dictionary Pairs for Classification

“…It is also worth noticing that the dictionary size, i.e., number of atoms, also has direct effect on the complexity of the compact representation of data. Thus, learning a good dictionary with the strong distinguishing power is crucial for the data representation and classification [1][2][3][4][5][6][7][8][9][10][11][12] [41][42][43][44][45].…”