Evaluation strategies for image understanding and retrieval

Abstract

We address evaluation of image understanding and retrieval large scale image data in the context of three evaluation projects. The first project is a comprehensive strategy for evaluating image retrieval algorithms and provides an open reference data set for doing so. The second project develops word prediction as a semantically relevant evaluation strategy, and applies it to the evaluation of of image processing methods for semantic image analysis. The third project evaluates words for suitability of their visual properties for use in an image annotation framework.

Tag-based social image retrieval: An empirical evaluation

“…Automatic image annotation refers to the task of assigning a few relevant keywords to an unannotated image to describe its visual content; the keywords are then indexed and used to retrieve images (Feng et al, 2004;Guillaumin et al, 2009;Jeon et al, 2003;Makadia et al, 2008;Yanai et al, 2005). These keywords are often derived from a well-annotated image collection, and the latter serves as training examples for automatic image annotation.…”

“…Moreover, in image annotation research, the keywords are carefully selected and the number of keywords is often very small. For instance, the number of keywords selected in the commonly used image annotation datasets, such as Corel5K, iarp tc12, and esp game datasets, ranges from 100 to 500 (Guillaumin et al, 2009;Jeon et al, 2003;Makadia et al, 2008;Yanai et al, 2005). The two larger datasets Corel30K and psu, containing 31K and 60K images, are annotated with 5,587 and 442 keywords, respectively (Carneiro et al, 2007).…”

“…Annotation-based image retrieval, on the other hand, enables users to formulate naturally semantic queries using textual keywords. In order to support this retrieval paradigm, many automatic image annotation techniques have been proposed, which assign a few relevant keywords to an unannotated image to describe its visual content for image indexing and retrieval (Feng, Manmatha, & Lavrenko, 2004;Guillaumin, Mensink, Verbeek, & Schmid, 2009;Jeon, Lavrenko, & Manmatha, 2003;Makadia, Pavlovic, & Kumar, 2008;Yanai, Shirahatti, Gabbur, & Barnard, 2005). The keywords are often derived from a well-annotated image collection and the number of keywords is often limited to a few hundred.…”

Discriminating image senses by clustering with multimodal features

“…They did use a larger set of image features and segmentation, however, we suspect that differences can rather be attributed to corpus type. In fact, (Yanai, Shirahatti, and Barnard, 2005) noted that human evaluators rated images obtained via a keyword retrieval method higher compared to image-based retrieval methods, which they relate to the importance of semantics for what humans regard as matching, and because pictorial semantics is hard to detect. (Cai et al, 2004) use similar methods to rank visual search results.…”

Multimedia object classification and retrieval

“…The inefficiencies of CBIR systems were reported early in the literature. In [YSGB05] it is stated that "current image retrieval methods are well off the required mark". When compared to the performance of current text retrieval systems, CBIR systems have a much lower success rate at retrieving relative images to a query.…”