Weakly Supervised Learning of Object Segmentations from Web-Scale Video

Abstract

Abstract. We propose to learn pixel-level segmentations of objects from weakly labeled (tagged) internet videos. Specifically, given a large collection of raw YouTube content, along with potentially noisy tags, our goal is to automatically generate spatiotemporal masks for each object, such as "dog", without employing any pre-trained object detectors. We formulate this problem as learning weakly supervised classifiers for a set of independent spatio-temporal segments. The object seeds obtained using segment-level classifiers are further refined using graphcuts to generate high-precision object masks. Our results, obtained by training on a dataset of 20,000 YouTube videos weakly tagged into 15 classes, demonstrate automatic extraction of pixel-level object masks. Evaluated against a ground-truthed subset of 50,000 frames with pixel-level annotations, we confirm that our proposed methods can learn good object masks just by watching YouTube.

Supervoxel-Consistent Foreground Propagation in Video

“…Methods for this "weakly supervised" setting attempt to learn an object model from ambiguously labeled exemplars [15,23,28,30]. This is very different from the propagation problem we tackle; our method gets only one video at a time and cannot benefit from cross-video appearance sharing.…”

Discriminative Segment Annotation in Weakly Labeled Video

“…The second scenario, which we term inductive segment annotation (ISA), is studied in [11]. In this setting, a segment classifier is trained using a large quantity of weakly labeled segments from both positively-and negatively-tagged videos.…”

“…We make publicly available segment-level annotations for a subset of the Prest et al dataset [20] and show convincing results. We also show state-of-the-art results on Hartmann et al's more difficult, large-scale object segmentation dataset [11]. …”

“…However, since tags are not spatially or temporally localized within the video, such videos cannot be directly exploited for training traditional supervised recognition systems. This has stimulated significant recent interest in methods that learn localized concepts under weak supervision [11,16,20,25]. In this paper, we examine the problem of generating pixel-level concept annotations for weakly labeled video.…”

Efficient Object Localization and Segmentation in Weakly Labeled Videos

“…Weakly supervised video segmentation methods [10][11][12] are proposed to curtail the need of pixel-level labeled training video data. Our proposed method is inspired by this line of research.…”