Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary

Abstract

We describe a model of object recognition as machine translation. In this model, recognition is a process of annotating image regions with words. Firstly, images are segmented into regions, which are classified into region types using a variety of features. A mapping between region types and keywords supplied with the images, is then learned, using a method based around EM. This process is analogous with learning a lexicon from an aligned bitext. For the implementation we describe, these words are nouns taken from a large vocabulary. On a large test set, the method can predict numerous words with high accuracy. Simple methods identify words that cannot be predicted well. We show how to cluster words that individually are difficult to predict into clusters that can be predicted well -for example, we cannot predict the distinction between train and locomotive using the current set of features, but we can predict the underlying concept. The method is trained on a substantial collection of images. Extensive experimental results illustrate the strengths and weaknesses of the approach.

Large Scale Online Learning of Image Similarity through Ranking

“…The mapping of the descriptors to discrete indexes is performed according to a codebook C, which is typically learned from the local descriptors of the training images through kmeans clustering (Duygulu et al, 2002, Jeon and Manmatha, 2004, Quelhas et al, 2005. The assignment of the weight p i of visterm i in image p is as follows:…”

Exploring SVM for Image Annotation in Presence of Confusing Labels

“…These are Corel5k [4], ESP Game [20] and IAPRTC-12 [7]. While Corel-5k has become the de-facto dataset in this domain, the other two datasets are very challenging with significant diversity among their samples.…”

“…cars, tracks, prototype grass, flowers, petals sky, grass, plane, lion tree, grass, tiger, park Figure 1: Example images from the Corel-5k dataset [4] and corresponding ground-truth labels. First image is an example of incomplete-labeling (tagged with "car" but not with "vehicle"); second image is an example of label-ambiguity (tagged with "flowers", though "blooms" would also have been equally correct); & third and fourth images are examples of structural-overlap ("lion" and "tiger" are two different but structurally related labels).…”

“…different labels, structurally they are very similar. Figure 1 shows such examples from Corel5k dataset [4]. All these issues combinedly give rise to the existence of sets of confusing labels within a vocabulary.…”

“…Among the image annotation models being proposed in the past, generative or nearestneighbour (NN)-based models [5,8,11,19,23] have particularly been shown to be successful for large vocabulary datasets such as Corel-5k [4], ESP Game [20] and IAPRTC-12 [7]. The reason behind this is that in NN-based models, given a sample, the labels that are not present in the ground-truth of its neighbouring samples are simply ignored, rather than being considered as negative.…”