Context-based multi-label image annotation

Abstract

This paper presents a novel context-based keyword propagation method for automatic image annotation. We follow the idea of keyword propagation and formulate image annotation as a multi-label learning problem, which is further resolved efficiently by linear programming. In this way, our method can exploit the context between keywords during keyword propagation. Unlike the popular relevance models that treat each keyword independently, our method can simultaneously propagate multiple keywords (i.e. labels) from the training images to the test images using their similarities. Moreover, we present a new 2D string kernel, called spatial spectrum kernel, to take into account another type of context when defining the similarity between images for keyword propagation. Each image is first denoted as a 2D sequence of visual keywords which are obtained through dividing images into blocks and then clustering these blocks, and a spatial spectrum kernel is then proposed to measure the 2D sequence similarity based on shared occurrences of s-length 1D subsequences through decomposing each 2D sequence into two parallel 1D sequences (i.e. the row-wise and column-wise ones). That is, we incorporate the context between visual keywords into the similarity between images (i.e. 2D sequences) used for keyword propagation. Experiments on two standard image databases demonstrate that the proposed method for automatic image annotation outperforms the state-of-the-art methods.

Image annotation with high-level words using generalised attributes

“…Each word might have several dictionary senses that are visually distinct, which is known as the 'visual polysemous' property. Therefore, image annotation can be transformed into a multi-class classification task, which can be solved using support vector machines (SVMs) or multi-label learning algorithms (Lu et al 2009) and multi-instance learning algorithms (Zhou and Zhang 2006). The co-occurrence probability between regional image features and concepts can be estimated to produce annotations for images.…”

Multi-label learning by Image-to-Class distance for scene classification and image annotation

“…For the application of image annotation task, we use MSRC dataset for evaluation, which is a new dataset but widely used in recent years [8,10,13,15,18]. This dataset consists of 591 images assigned to 21 predefined labels (As suggested by MSRC, the label "horse" and "mountain" are ignored due to a limited number of labeled images).…”

“…In addition, we compare our method to related methods like MBRM [4] and SSK [10] by F1 score, since this evaluation metric is commonly used in automatic image annotation studies and has been used by others in this dataset. It measures the quality of retrieved relevant images with each keyword, which is treated as class label here.…”

“…Our methods are compared with MBRM [4] and SSK combined with CBKP [10]. Same as [10], the top 3 keywords in the rank list is taken as automatic annotation of a test image.…”

Content-Based Image Retrieval System for Real Images

“…Therefore, extracting only the highlight of an image (or the main object of an image) is essential to improve classification accuracy. Here, an image method called JSEG [21] is used to segment the objects in an image. The JSEG method consists of two steps, color space quantization and spatial segmentation.…”