Video search reranking via information bottleneck principle

Abstract

We propose a novel and generic video/image reranking algorithm, IB reranking, which reorders results from text-only searches by discovering the salient visual patterns of relevant and irrelevant shots from the approximate relevance provided by text results. The IB reranking method, based on a rigorous Information Bottleneck (IB) principle, finds the optimal clustering of images that preserves the maximal mutual information between the search relevance and the high-dimensional low-level visual features of the images in the text search results. Evaluating the approach on the TRECVID 2003-2005 data sets shows significant improvement upon the text search baseline, with relative increases in average performance of up to 23%. The method requires no image search examples from the user, but is competitive with other state-of-the-art example-based approaches. The method is also highly generic and performs comparably with sophisticated models which are highly tuned for specific classes of queries, such as named-persons. Our experimental analysis has also confirmed the proposed reranking method works well when there exist sufficient recurrent visual patterns in the search results, as often the case in multi-source news videos.

An Adaptive Query Prototype Modeling Method for Image Search Reranking

“…The images reranking methods can be classified into that of classification based [12,25], clustering based [2,3,4,7,14], graph based [8,16], and learning to rerank [22,26,27].…”

“…Clustering based methods, such as [2,3,4,7,14], have similar several integral components. Specifically, in [2], each image is segmented into similar regions or "blobs".…”

“…Lastly, initial rank images are reranked based on similarity clusters. And, most of clustering-based methods [7] [2] use global features like color or texture features. However, the foreground parts in the image will have more influence to the relevance score of the image to the query keyword.…”

Query-Adaptive Fusion for Multimodal Search

“…Another fusion method that we have explored in recent work [51], [52] is a reranking approach, where the results of one search method are taken as a hypothesis of the actual desired ranking of the documents in the search set. This initial list of results is then mined in another modality to infer recurrent characteristics of relevant documents which can be used to reorder and refine the results.…”

“…This initial list of results is then mined in another modality to infer recurrent characteristics of relevant documents which can be used to reorder and refine the results. Both [51] and [52] are applied in the TRECVID broadcast news video domain. In [51], the initial search is taken from a text search over the speech recognition transcripts and the results are reranked by taking the top-returned documents to be pseudo-positive and pseudo-negative examples are sampled from elsewhere in the list.…”