A reranking approach for context-based concept fusion in video indexing and retrieval

Abstract

We propose to incorporate hundreds of pre-trained concept detectors to provide contextual information for improving the performance of multimodal video search. The approach takes initial search results from established video search methods (which typically are conservative in usage of concept detectors) and mines these results to discover and leverage co-occurrence patterns with detection results for hundreds of other concepts, thereby refining and reranking the initial video search result. We test the method on TRECVID 2005 and 2006 automatic video search tasks and find improvements in mean average precision (MAP) of 15%-30%. We also find that the method is adept at discovering contextual relationships that are unique to news stories occurring in the search set, which would be difficult or impossible to discover even if external training data were available.

Computer Vision for Lifelogging

“…Detection refinement or adjustment methods [43,44,45,46,47] represent a stream of post-processing method which can enhance detection scores obtained from individual detectors, allowing independent and specialized classification techniques to be leveraged for each concept. In this section, we introduce an approach which can exploit the inter-concept relationships implicity from concept detection results of C N×M in order to provide better quality semantic indexing.…”

A generic framework for semantic video indexing based on visual concepts/contexts detection

“…Therefore, much new research has involved the exploration of the semantic knowledge among concepts for video indexing. They particularly aim to develop a context-based concept fusion (CBCF) framework to enhance the concept detection results [16,17,20,45,54,57]. These approaches fall into two categories.…”

“…The second category is based on learning techniques [16,20,45]. In [45], the contextual relationship is modeled by SVM.…”

Re-ranking for Multimedia Indexing and Retrieval

“…Classification-based re-ranking [3]: the initial results of a baseline system are used to discover the co-occurrence patterns between the target semantics and extracted features. This is very similar to "learning to rank" [4], which is based on training a ranking model which can precisely predict the ranking lists in the dataset.…”

“…This is very similar to "learning to rank" [4], which is based on training a ranking model which can precisely predict the ranking lists in the dataset. In [3] the authors used the top-ranked and bottom-ranked samples respectively, as pseudo-positive and pseudonegative examples to train a new classification model for ranking, and the classification margin for a target concept is regarded as its (new) re-ranked. The use of SVM as the classification model, leads to the method called RankSVM [4].…”