Bridging the Gap: Query by Semantic Example

Abstract

Abstract-A combination of query-by-visual-example (QBVE) and semantic retrieval (SR), denoted as query-by-semantic-example (QBSE), is proposed. Images are labeled with respect to a vocabulary of visual concepts, as is usual in SR. Each image is then represented by a vector, referred to as a semantic multinomial, of posterior concept probabilities. Retrieval is based on the query-by-example paradigm: the user provides a query image, for which 1) a semantic multinomial is computed and 2) matched to those in the database. QBSE is shown to have two main properties of interest, one mostly practical and the other philosophical. From a practical standpoint, because it inherits the generalization ability of SR inside the space of known visual concepts (referred to as the semantic space) but performs much better outside of it, QBSE produces retrieval systems that are more accurate than what was previously possible. Philosophically, because it allows a direct comparison of visual and semantic representations under a common query paradigm, QBSE enables the design of experiments that explicitly test the value of semantic representations for image retrieval. An implementation of QBSE under the minimum probability of error (MPE) retrieval framework, previously applied with success to both QBVE and SR, is proposed, and used to demonstrate the two properties. In particular, an extensive objective comparison of QBSE with QBVE is presented, showing that the former significantly outperforms the latter both inside and outside the semantic space. By carefully controlling the structure of the semantic space, it is also shown that this improvement can only be attributed to the semantic nature of the representation on which QBSE is based.Index Terms-Content-based image retrieval, Gaussian mixtures, image similarity, multiple instance learning, query by example, semantic retrieval, semantic space.

Image retrieval using query by contextual example

“…By exploiting the statistical structure of these ambiguities, a QBSE system is able to perform inferences at an higher level of abstraction, and significantly outperform QBVE systems. This has been confirmed by various recent studies, which have shown that QBSE systems can generalize much better than their QBVE counterparts [13].…”

“…For brevity, we limit the discussion to the implementation details of contextual level. The visual and semantic levels are those proposed in [19] and [13], where they were shown to achieve better performance than a number of other state of the art image retrieval systems.…”

“…Image retrieval results are presented on two public datasets -1) Natural15: 15-Natural scene classes [6] and 2) Corel15: 15-Corel stock photo CD's, used in [13] for QBSE. The Natural15 dataset contains 200-400 images per class with an average image size of 300×250px.…”

“…To address this limitation, various authors have proposed an alternative query-by-example strategy which extends QBVE to the semantic domain [17,18,10,14,13,8]. This strategy, commonly referred to as query-by-semantic-example (QBSE), formulates image retrieval as a two stage process.…”

Multimedia translation for linking visual data to semantics in videos

“…While given a single video story learning the links between visual appearances of keyframes and speech transcript words is not possible, the concurrent occurrences in large number of available video stories provide that information learned, they can be used to predict labels for the regions (region labeling) as an alternative to large-scale object recognition. They can also be used in a setting similar to the query by semantic example method as proposed by [67].…”