Image Annotation Using Metric Learning in Semantic Neighbourhoods

Abstract

Abstract. Automatic image annotation aims at predicting a set of textual labels for an image that describe its semantics. These are usually taken from an annotation vocabulary of few hundred labels. Because of the large vocabulary, there is a high variance in the number of images corresponding to different labels ("class-imbalance"). Additionally, due to the limitations of manual annotation, a significant number of available images are not annotated with all the relevant labels ("weaklabelling"). These two issues badly affect the performance of most of the existing image annotation models. In this work, we propose 2PKNN, a two-step variant of the classical K-nearest neighbour algorithm, that addresses these two issues in the image annotation task. The first step of 2PKNN uses "image-to-label" similarities, while the second step uses "image-to-image" similarities; thus combining the benefits of both. Since the performance of nearest-neighbour based methods greatly depends on how features are compared, we also propose a metric learning framework over 2PKNN that learns weights for multiple features as well as distances together. This is done in a large margin set-up by generalizing a well-known (single-label) classification metric learning algorithm for multi-label prediction. For scalability, we implement it by alternating between stochastic sub-gradient descent and projection steps. Extensive experiments demonstrate that, though conceptually simple, 2PKNN alone performs comparable to the current state-of-the-art on three challenging image annotation datasets, and shows significant improvements after metric learning.

Exploring SVM for Image Annotation in Presence of Confusing Labels

“…We use the same evaluation criteria as being used by previous methods [6,8,11,19,23]. Given a new sample, first we compute the score for each label using the corresponding classifier, and then assign it the five top-scoring labels.…”

“…We also compare with two SVM-based models [1,9]. In [9], we use Table 3: Performance comparison between the current state-of-the-art [19] and the best results of this work.…”

“…We use three datasets popular in the image annotation task [5,8,11,19]. These are Corel5k [4], ESP Game [20] and IAPRTC-12 [7].…”

“…This has applications in several tasks such as image retrieval, object recognition, robot navigation, etc. Hence this has emerged as an important research area during the last decade [2,5,6,8,11,19,23]. In annotation datasets with large vocabularies of few hundred or more labels, there exist three practical issues: (a) Incomplete-labeling: The training samples are not exhaustively tagged with all relevant labels from vocabulary.…”

“…Among the image annotation models being proposed in the past, generative or nearestneighbour (NN)-based models [5,8,11,19,23] have particularly been shown to be successful for large vocabulary datasets such as Corel-5k [4], ESP Game [20] and IAPRTC-12 [7]. The reason behind this is that in NN-based models, given a sample, the labels that are not present in the ground-truth of its neighbouring samples are simply ignored, rather than being considered as negative.…”