The segmented and annotated IAPR TC-12 benchmark

Abstract

With the increasing storage of images worldwide, automatic image annotation has become a very active and relevant research area, however, it still lacks a benchmark specifically designed for this task, and in particular for region-level annotation. In this report we introduce the segmented and annotated IAPR-TC12 benchmark, an extended resource for the evaluation of automatic image annotation (AIA) methods. We present a methodology for the manual segmentation and annotation of the images in this collection. The goal of this methodology is to obtain reliable ground truth data for benchmarking AIA and related tasks. For annotation, an ad-hoc vocabulary is defined and hierarchically organized. This hierarchy proved to be very useful for obtaining objective and structured annotations. Also, a soft measure for the evaluation of annotation performance is proposed, based on this hierarchy. Statistics on the segmentation and annotation processes give evidence of the reliability of the proposed approach. Visual attributes and spatial relations are also extracted from regions in segmented images. The latter feature will promote research on the use of (spatial) contextual information for AIA and image retrieval. The extended collection is publicly available and can be used to evaluate a variety of tasks besides image annotation; this resource can also serve to study the use of automatic annotations for multimedia image retrieval; the latter is a distinctive feature of the collection because, although there are several image annotation benchmarks, there is currently no collection that can be used to effectively evaluate the performance of annotation methods in the task they are designed for (i.e. image retrieval). We outline several applications and raise important questions that might be answered with the annotated collection; motivating research in the areas of image segmentation, annotation and retrieval as well as on machine learning.

Understanding and predicting importance in images

“…The original IAPR TC-12 Benchmark [12] includes a free-text description for each image. Crucially, in its expansion (SAIAPR TC-12), each image is also segmented into constituent objects and labeled according to a set of (275) labels [5]. From here on we will refer to this dataset, and its descriptions as ImageCLEF.…”

Spatial Language Understanding with Multimodal Graphs using Declarative Learning based Programming

“…For example, the ontology for a specific object like Bush can be entity->landscape-nature-> vegetation->trees->bush. Given the image segments, the spatial relations between segments are automatically extracted in a pairwise exhaustive manner using the geometrical properties of the segments (Escalante et al, 2010). These relations are limited to relationships such as besides, disjoint, below, above, x-aligned, and y-aligned.…”

Multi-Instance Multi-Label Learning for Image Classification with Large Vocabularies

“…Method Table 1: AUC (± standard deviation) for MSRC V2 dataset IAPR TC-12 One recent benchmark in image annotation and retrieval is the IAPR TC-12 dataset [10]. It consists of 20,000 images each annotated with keywords from 274 categories.…”