Causal video object segmentation from persistence of occlusions

Abstract

Figure 1: Sample outcomes of our scheme: background c(x) = 0 (gray) and foreground layers c(x) = 1, c(x) = 2, c(x) = 3 indicated by , , respectively. On the far right, our algorithm correctly infers that the bag strap is in front of the woman's arm, which is in front of her trunk, which is in front of the background. Project page: http://vision.ucla.edu/cvos/ AbstractOcclusion relations inform the partition of the image domain into "objects" but are difficult to determine from a single image or short-baseline video. We show how long-term occlusion relations can be robustly inferred from video, and used within a convex optimization framework to segment the image domain into regions. We highlight the challenges in determining these occluder/occluded relations and ensuring regions remain temporally consistent, propose strategies to overcome them, and introduce an efficient numerical scheme to perform the partition directly on the pixel grid, without the need for superpixelization or other preprocessing steps.

Unsupervised Moving Object Detection via Contextual Information Separation

“…GT SFL [10] LMP [35] PDB [31] CVOS [34] FTS [28] ELM [22] Ours Figure 4: Qualitative Results: We qualitatively compare the performance of our approach with several state-of-the-art baselines as well as the Ground-Truth (GT) mask. Our prediction are robust to background clutter, large depth discontinuities and occlusions.…”

Object Discovery in Videos as Foreground Motion Clustering

“…Other works that build off this idea include formulating trajectory clustering as a multi-cut problem [23,24,25] or as a density peaks clustering [46], and detecting discontinuities in the trajectory spectral embedding [15]. More recent approaches include using occlusion relations to produce layered segmentations [43], combining piecewise rigid motions with pre-trained CNNs to merge the rigid motions into objects [7], and jointly estimating scene flow and motion segmentations [39]. We use pixel trajectories in a recurrent neural network to learn trajectory embeddings for motion clustering.…”

Minimum Delay Moving Object Detection

“…Dense motion has been computed across many frames in [22], but not for segmentation. Instead of batch processing to integrate motion cues over time, [31] integrates occlusion cues [24,30,19] over frames causally. Batch segmentation methods [16,12] may achieve a stronger motion signal at the expense of processing the whole batch.…”

“…We compare both frame-by-frame approaches [31], which integrates occlusion cues causally across time, and batch approaches [16,12] integrating motion cues across frames against our method. To apply [16,12] in an online approach, at each frame t, we segment frames 1 to t in batch, and then choose regions at time t that pass a relative area threshold as the detected regions.…”

“…If no regions pass the test, there is no detection, and frames 1 to t + 1 are considered, etc. Similarly, we threshold the result of [31] at each frame t to obtain a detection. We refer to the detection using [31] as CVOS+det, [16] as LVA+det, and [12] as Multicuts+det.…”