A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval

Abstract

Effective video retrieval is the result of an interplay between interactive query selection, advanced visualization of results, and a goal-oriented human user. Traditional interactive video retrieval approaches emphasize paradigms, such as query-by-keyword and query-by-example, to aid the user in the search for relevant footage. However, recent results in automatic indexing indicate that query-by-concept is becoming a viable resource for interactive retrieval also. We propose in this paper a new video retrieval paradigm. The core of the paradigm is formed by first detecting a large lexicon of semantic concepts. From there, we combine query-by-concept, query-by-example, query-by-keyword, and user interaction into the MediaMill semantic video search engine. To measure the impact of increasing lexicon size on interactive video retrieval performance, we performed two experiments against the 2004 and 2005 NIST TRECVID benchmarks, using lexicons containing 32 and 101 concepts, respectively. The results suggest that from all factors that play a role in interactive retrieval, a large lexicon of semantic concepts matters most. Indeed, by exploiting large lexicons, many video search questions are solvable without using query-by-keyword and query-by-example. In addition, we show that the lexicon-driven search engine outperforms all state-of-the-art video retrieval systems in both TRECVID 2004 and 2005.Index Terms-Benchmarking, concept learning, content analysis and indexing, interactive systems, multimedia information systems, video retrieval.

Content Based Video Retrieval Based on HDWT and Sparse Representation

“…On receiving a query, the same feature extraction method is applied on the query. Then, one of the mentioned similarity measures is calculated, and the retrieval results are shown according to the query (Snoek et al, 2007). This paper is organized as follows: In the following section, the proposed video retrieval method is described step by step.…”

Fast Re-ranking of Visual Search Results by Example Selection

“…Several ways have been explored to automatically annotate images based on co-occurrence of visual and textual information on the internet [13,15,17,18]. An example of a general-purpose large-scale system which learns new objects and relations from images is called NEIL [4].…”

Using Visual Context and Region Semantics for High-Level Concept Detection

“…In [56], the authors train separate shape detectors using a shape alphabet, which is actually a dictionary of curve fragments. A lexicon-driven approach, used within an interactive video retrieval system, is presented by Snoek et al in [66] and [65]. Finally, in [34], an approach for texture and object recognition is presented.…”