Multi-view Object Segmentation in Space and Time

Abstract

In this paper, we address the problem of object segmentation in multiple views or videos when two or more viewpoints of the same scene are available. We propose a new approach that propagates segmentation coherence information in both space and time, hence allowing evidences in one image to be shared over the complete set. To this aim the segmentation is cast as a single efficient labeling problem over space and time with graph cuts. In contrast to most existing multi-view segmentation methods that rely on some form of dense reconstruction, ours only requires a sparse 3D sampling to propagate information between viewpoints. The approach is thoroughly evaluated on standard multiview datasets, as well as on videos. With static views, results compete with state of the art methods but they are achieved with significantly fewer viewpoints. With multiple videos, we report results that demonstrate the benefit of segmentation propagation through temporal cues.

Boosting objectness: Semi-supervised learning for object detection and segmentation in multi-view images

“…We obtained five datasets from two stateof-the-art methods: BUSTE dataset from [7] for qualitative evaluation; COUCH, TEDDY, CHAIR1, CAR from [9] which we use for both qualitative and quantitative evaluation. We use the same evaluation metric as [9,11], computing the intersection over union to measure the segmentation quality. Fig.…”

“…Yet fully automatic methods remain useful in scenarios where the human in the loop is impractical, such as large scale usergenerated content processing. Prior automatic algorithms [5,6,7,8,9,10,11] typically draw upon geometric constraints implicit within multi-view scenario. These methods either simultaneously derive segmentation whilst performing costly visual hull estimation [5], or rely on the color distributions of the object and background being very different [6,7,10,11], or require the fixation of cameras on the object [6,8], or dense depth recovery of the whole scene [9].…”

3D reconstruction for featureless scenes with curvature hints

“…Similar to [33], they use a GMM model and an iterative process for a posteriori estimation (MAP) of the classification variables. Sparse 3D samples are also used by Djelouah et al [13], where the problem of multi-view segmentation is extended on the time dimension to support multi-view video segmentation and superpixels are used to reduce the computational complexity.…”

Consistent 3D Background Model Estimation from Multi-viewpoint Videos

“…The latter can be fully automatic, under the assumptions of [23], that is if the foreground is entirely visible in all cameras, and by operating on pixels obtains more accurate boundaries. Djelouah et al [6,7] proposed an approach that links multiple views via an MRF and is able to handle videos as input, and not just individual frames. These methods achieve spatially consistent segmentation but do not estimate depths for the background.…”