Sparse Multi-View Consistency for Object Segmentation

Abstract

Multiple view segmentation consists in segmenting objects simultaneously in several views. A key issue in that respect and compared to monocular settings is to ensure propagation of segmentation information between views while minimizing complexity and computational cost. In this work, we first investigate the idea that examining measurements at the projections of a sparse set of 3D points is sufficient to achieve this goal. The proposed algorithm softly assigns each of these 3D samples to the scene background if it projects on the background region in at least one view, or to the foreground if it projects on foreground region in all views. Second, we show how other modalities such as depth may be seamlessly integrated in the model and benefit the segmentation. The paper exposes a detailed set of experiments used to validate the algorithm, showing results comparable with the state of art, with reduced computational complexity. We also discuss the use of different modalities for specific situations, such as dealing with a low number of viewpoints or a scene with color ambiguities between foreground and background.

Fusing Depth and Silhouette for Scanning Transparent Object with RGB-D Sensor

“…Our method falls on the other stream: segmentation on images [15][16][17]. Zhang et al [15] let multiple color images share the GMM color model of foreground and background and propagate the segmentation cues across views by silhouette consistency which is induced by depth.…”

Model-Based Outdoor Performance Capture

“…A key limitation of many existing performance capture methods is their dependency on explicit background segmentation for accurate silhouette alignment, an errorprone step which hinders their usage in uncontrolled environments. Progress has been made by multi-view segmentation [51,16], joint segmentation and reconstruction [45,22,8,32,33,14], and also aided by propagation of a manual initialization [23,53,46]. In uncontrolled environments the obtained segmentation is still noisy, enabling only skeleton pose [23] and rather coarse 3D reconstructions [32,33].…”

Consistent 3D Background Model Estimation from Multi-viewpoint Videos

“…The latter can be fully automatic, under the assumptions of [23], that is if the foreground is entirely visible in all cameras, and by operating on pixels obtains more accurate boundaries. Djelouah et al [6,7] proposed an approach that links multiple views via an MRF and is able to handle videos as input, and not just individual frames. These methods achieve spatially consistent segmentation but do not estimate depths for the background.…”