Joint visual-text modeling for automatic retrieval of multimedia documents

Abstract

In this paper we describe a novel approach for jointly modeling the text and the visual components of multimedia documents for the purpose of information retrieval(IR). We propose a novel framework where individual components are developed to model different relationships between documents and queries and then combined into a joint retrieval framework. In the state-of-the-art systems, a late combination between two independent systems, one analyzing just the text part of such documents, and the other analyzing the visual part without leveraging any knowledge acquired in the text processing, is the norm. Such systems rarely exceed the performance of any single modality (i.e. text or video) in information retrieval tasks. Our experiments indicate that allowing a rich interaction between the modalities results in significant improvement in performance over any single modality. We demonstrate these results using the TRECVID03 corpus, which comprises 120 hours of broadcast news videos. Our results demonstrate over 14% improvement in IR performance over the best reported textonly baseline and ranks amongst the best results reported on this corpus.

YouTube Scale, Large Vocabulary Video Annotation

“…One might expect that these words would be predicted entirely by co-occurrence probabilities with their counterparts. We find that the combined model using both text and vision almost always produces the best results, an observation that is shared with other works in image and video annotation [21,19] Table 2 Top-20 annotation accuracy of selected pairs of words which naturally co-occur together.…”

A discrete direct retrieval model for image and video retrieval

“…The first one is a standard Corel image set which contains 5000 images widely used for comparing results. The second one is the large scale data set consisting of the entire TRECVID 2003 development dataset and feature set used by [11].…”

“…For comparison the best published retrieval results we know of on the same data set are 0.31 (SML [6]) and 0.30 (NCRM [14]). On a TRECVID3 dataset [11] the corresponding numbers are 0.152 and 0.158 for the diswhere they do not work well for long queries crete MRF model and the NCRM model respectively. The discrete MRF takes 90s for all queries while NCRM takes 6.8 hrs.…”

Visual Semantic Search: Retrieving Videos via Complex Textual Queries

“…Iyengar et al [11] proposed a probabilistic model that relates words and image parts through an intermediate layer that captures common concepts. Models in this category usually rely on strong assumptions, e.g.…”