Using heterogeneous annotation and visual information for the benchmarking of image retrieval systems

Abstract

Many image retrieval systems, and the evaluation methodologies of these systems, make use of either visual or textual information only. Only few combine textual and visual features for retrieval and evaluation. If text is used, it is often relies upon having a standardised and complete annotation schema for the entire collection. This, in combination with high-level semantic queries, makes visual/textual combinations almost useless as the information need can often be solved using just textual features. In reality, many collections do have some form of annotation but this is often heterogeneous and incomplete. Web-based image repositories such as FlickR even allow collective, as well as multilingual annotation of multimedia objects.This article describes an image retrieval evaluation campaign called ImageCLEF. Unlike previous evaluations, we offer a range of realistic tasks and image collections in which combining text and visual features is likely to obtain the best results. In particular, we offer a medical retrieval task which models exactly the situation of heterogenous annotation by combining four collections with annotations of varying quality, structure, extent and language. Two collections have an annotation per case and not per image, which is normal in the medical domain, making it difficult to relate parts of the accompanying text to corresponding images. This is also typical of image retrieval from the Web in which adjacent text does not always describe an image. The ImageCLEF benchmark shows the need for realistic and standardised datasets, search tasks and ground truths for visual information retrieval evaluation.

A Framework for Web Science

“…A further important open issue is the interoperability of Semantic Web technologies with non-RDF-based metadata such as EXIF metadata on JPEGS or the informal image tags created in Flickr [279]. Further work is required on the relationship between human image retrieval requirements and the possibilities of automation [156,206], including a deeper understanding of the relative capabilities of folksonomies and ontologies (see Sections 3.1.4-3.1.5).…”

All‐visual retrieval: How people search and respond to an affect‐driven visual information retrieval system

“…(Smith, 1998, p.112) In brief, then, there remains a need to understand how people actually interact with image-based systems. While there are advances in supporting VIR (Ward & Reinhard, 2000) and while Boolean searches remain a potential source for testing VIR at the semantic level (Zhang, 2007), the inclusion of heterogeneous sources of data (Müller et al, 2006;Benoît, 2008) and searching by appearance (e.g. Siggelkow, n.d.), projects or computational-or human-identified aspects remains a challenge.…”

Entropy-Guided Assessment of Image Retrieval Systems: Advancing Grouped Precision as an Evaluation Measure for Relevant Retrievability

“…The goal of a benchmark is to compare different systems on a set of test images database. An exhaustive survey on this topic can be found for example in [19], [21] and [20].…”