Multimedia semantic indexing using model vectors

Abstract

In this paper we propose a novel method for multimedia semantic indexing using model vectors. Model vectors provide a semantic signature for multimedia documents by capturing the detection of concepts broadly across a lexicon using a set of independent binary classifiers. While recent techniques have been developed for detecting simple generic concepts such as indoors, outdoors, nature, manmade, faces, people, speech, music, and so forth [1], these labels directly support only a small number of queries. Model vectors address the problem of answering queries for which relationships to specific concepts is either unknown or indirect by developing a basis across across the lexicon. In the simplest case, each model vector dimension corresponds to the confidence score by which a corresponding concept from the lexicon is detected. However, we show how other information such as relevance, reliability and concept correlation can also be incorporated. Overall, the model vectors can be used in a variety of methods for multimedia indexing, including model-based retrieval, relevance feedback searching and concept querying. In this paper, we present the model vector method and study different strategies for computing and comparing model vectors. We empirically evaluate the retrieval effectiveness of the model vector approach compared to other search methods in a large video retrieval testbed.

A generic framework for semantic video indexing based on visual concepts/contexts detection

“…After that, an SVM is learned in order to refine the detection of the original concepts. Although some performance improvement is reported in [16,45], there are major drawbacks. First, these approaches are fully supervised and require explicit knowledge of the target semantic concept and ground-truth labels such as the ontology hierarchy and the Bayesian networks (manually constructed in most cases).…”

“…Therefore, much new research has involved the exploration of the semantic knowledge among concepts for video indexing. They particularly aim to develop a context-based concept fusion (CBCF) framework to enhance the concept detection results [16,17,20,45,54,57]. These approaches fall into two categories.…”

“…The second category is based on learning techniques [16,20,45]. In [45], the contextual relationship is modeled by SVM.…”

“…In [45], the contextual relationship is modeled by SVM. Firstly, the Discriminative Model Fusion (DMF) method generates a model vector aggregating the detection score of all the individual detectors.…”

Prosemantic Features for Content-Based Image Retrieval

“…One of the first works that try to bring semantic information under the same model vector paradigm used in query-by-example systems is [15]. Semantic information is learned directly from the image content and forms a vector of semantic weights.…”