Video Motion for Every Visible Point

Abstract

Dense motion of image points over many video frames can provide important information about the world. However, occlusions and drift make it impossible to compute long motion paths by merely concatenating optical flow vectors between consecutive frames. Instead, we solve for entire paths directly, and flag the frames in which each is visible. As in previous work, we anchor each path to a unique pixel which guarantees an even spatial distribution of paths. Unlike earlier methods, we allow paths to be anchored in any frame. By explicitly requiring that at least one visible path passes within a small neighborhood of every pixel, we guarantee complete coverage of all visible points in all frames. We achieve state-of-the-art results on real sequences including both rigid and non-rigid motions with significant occlusions.

Occlusion-aware video registration for highly non-rigid objects

“…Here, [16] presents a variational approach that estimates visibility maps for each pixel, using two reference frames and hard subspace constraints of the pixel displacements. The same authors improved upon this method in [17] by introducing a different strategy for visibility labeling and even more reference images, but the computational complexity was very high. Furthermore, the authors of [17] explicitly mention that their method can not deal with the highly deformable motion of the waving flag sequence of [10,8] that we use in our evaluation.…”

“…The same authors improved upon this method in [17] by introducing a different strategy for visibility labeling and even more reference images, but the computational complexity was very high. Furthermore, the authors of [17] explicitly mention that their method can not deal with the highly deformable motion of the waving flag sequence of [10,8] that we use in our evaluation. In general, the referenced literature contains two general concepts for occlusion modelling and correction.…”

“…The latter is also robust against large self-occlusions by learning a model for the image distortion and is included in our evaluation. In the case of MFOF, contributions to explicit occlusion detection and handling are [16,17]. Here, [16] presents a variational approach that estimates visibility maps for each pixel, using two reference frames and hard subspace constraints of the pixel displacements.…”

“…This wrong optical flow fields introduces noise for correspondence computations, see Figure 1. Current MFOF approaches only consider occlusion estimation under rather rigid motion with only slightly nonrigid deformations [16,17] that is not reliable under high non-rigidity [17]. Therefore, we introduce an explicit occlusion detection, tracking and handling that is efficient and gives consistent results in the presence of highly non-rigid deformations.…”

3D Semantic Trajectory Reconstruction from 3D Pixel Continuum

“…A core challenge of motion segmentation lies in fragmented nature of trajectories caused by tracking failure (occlusion, drifting, and motion blur). Embedding trajectories into low dimensional space has been used to robustly measure trajectory distance in the presence of missing data without pre-trained models [9,13,16,29], and 2D trajectories can be decomposed into 3D camera motion and deformable object models [27,33,38]. Visual semantics learned by object recognition frameworks provides stronger cues to cluster trajectories [21,22,36].…”