Zero-shot video retrieval using content and concepts

Abstract

Recent research in video retrieval has been successful at finding videos when the query consists of tens or hundreds of sample relevant videos for training supervised models. Instead, we investigate unsupervised zero-shot retrieval where no training videos are provided: a query consists only of a text statement. For retrieval, we use text extracted from images in the videos, text recognized in the speech of its audio track, as well as automatically detected semantically meaningful visual video concepts identified with widely varying confidence in the videos. In this work we introduce a new method for automatically identifying relevant concepts given a text query using the Markov Random Field (MRF) retrieval framework. We use source expansion to build rich textual representations of semantic video concepts from large external sources such as the web. We find that concept-based retrieval significantly outperforms text based approaches in recall. Using an evaluation derived from the TRECVID MED'11 track, we present early results that an approach using multi-modal fusion can compensate for inadequacies in each modality, resulting in substantial effectiveness gains. With relevance feedback, our approach provides additional improvements of over 50%.

Zero-Example Event Search using MultiModal Pseudo Relevance Feedback

“…The second method is the Rocchio PRF, where parameters were set to a = 1.0, b = 0.8, c = 0.5 [9]. The third method is the Relevance Model (R-M) [15,4,18], where the variant with the i.i.d. assumption was used.…”

“…Since a document's true relevance judgment is unavailable, the top-ranked and bottom-ranked documents in the retrieved list are used to approximate the relevant and irrelevant documents. In the language model, PRF is usually performed with a Relevance Model (RM) [15,4,18]. The idea is to estimate the probability of a word in the relevance model, and feed the probability back to smooth the query likelihood in the language model.…”

“…Search a pseudo label set y (t) including k + pseudo-positives and k − pseudo-negatives; 4 Training a joint model on y (t) using both high-level features and low-level features;…”

“…Generally, a basic 0Ex pipeline consists of the following stages. The first stage is called query generation, in which the system converts the input event kit description into a set of queries [24,30,4]. For example, Figure 1 shows the example query generated from the words in the event kit description of "Birthday party".…”

Visual Semantic Search: Retrieving Videos via Complex Textual Queries

“…The importance of concepts have also been recognized by other researchers [25]. Since then, a series of methods have been developed to improve video concept detection [1,21,5]. It is important to note that most current work on concept-based retrieval focuses on mining characteristic concepts or improving the performance of concept detectors.…”