Discover and Learn New Objects from Documentaries

Abstract

Despite the remarkable progress in recent years, detecting objects in a new context remains a challenging task. Detectors learned from a public dataset can only work with a fixed list of categories, while training from scratch usually requires a large amount of training data with detailed annotations. This work aims to explore a novel approach -learning object detectors from documentary films in a weakly supervised manner. This is inspired by the observation that documentaries often provide dedicated exposition of certain object categories, where visual presentations are aligned with subtitles. We believe that object detectors can be learned from such a rich source of information. Towards this goal, we develop a joint probabilistic framework, where individual pieces of information, including video frames and subtitles, are brought together via both visual and linguistic links. On top of this formulation, we further derive a weakly supervised learning algorithm, where object model learning and training set mining are unified in an optimization procedure. Experimental results on a real world dataset demonstrate that this is an effective approach to learning new object detectors.

End-to-End Learning of Visual Representations from Uncurated Instructional Videos

“…In contrast, in this work no manually annotated visual data is involved at any stage of our approach. To avoid labelling visual data, several approaches have leveraged audio transcripts obtained from narrated videos using automatic speech recognition (ASR) as a way to supervise video models for object detection [3,15,54], captioning [33,69], classification [2,42,47,86], summarization [57] or retrieval [50] using large-scale narrated video datasets such as How2 [65] or HowTo100M [50]. Others [10,30] have investigated learning from narrated videos by directly using the raw speech waveform instead of generating transcriptions.…”

Learning to Find Common Objects Across Few Image Collections

“…Multiple instance learning (MIL) [7,33] methods have been used for learning weakly supervised tasks such as object localization (WSOL) [25,8,53,41]. In a standard MIL framework, instance labels in each positive bag are treated as hidden variables with the constraint that at least one of them should be positive.…”

Learning to discover and localize visual objects with open vocabulary

“…[3,13] focus on separating distinguishable audio and video objects simultaneously. [6] learn to associate tracklets with words in documentary subtitles. Most of these multi-modal methods primarily focus on captioning or retrieval tasks, while our main focus is localization.…”

“…We believe an essential step to scale up to millions of object classes is to use abundant and labor-free web data. One pioneering work is from Chen et al [6] which learns to discover and localize new objects from documentary videos by associating subtitles to video tracklets. There is also work to associate phrases in the caption to its visually depicted objects in the image [33,20].…”

“…These show encouraging re-sults of using noisy and weakly annotated multi-modal data for learning visual concepts and localizing objects. However, [6] requires video tracklets as input which is limiting, and [33] requires image captions at prediction time as well, and thereby can not be used as an object detector.…”