Love Thy Neighbors: Image Annotation by Exploiting Image Metadata

Abstract

Some images that are difficult to recognize on their own may become more clear in the context of a neighborhood of related images with similar social-network metadata. We build on this intuition to improve multilabel image annotation. Our model uses image metadata nonparametrically to generate neighborhoods of related images using Jaccard similarities, then uses a deep neural network to blend visual information from the image and its neighbors. Prior work typically models image metadata parametrically; in contrast, our nonparametric treatment allows our model to perform well even when the vocabulary of metadata changes between training and testing. We perform comprehensive experiments on the NUS-WIDE dataset, where we show that our model outperforms state-of-the-art methods for multilabel image annotation even when our model is forced to generalize to new types of metadata.

Diverse Image Annotation

“…Such networks learn very promising features for many tasks, such as image classification [15] and object detection [23]. Global CNN-based image features have been used for image annotation too [13]; however, some recent work [10] [27] learns local features for detected bounding boxes, so as to extract more discriminative object-centric features rather than from background. The second direction focuses on exploring and exploiting tag correlations.…”

“…As such, image annotation is treated as a multi-label learning problem, where tag correlations play a key role. Most common tag correlations involve tag-level smoothness [30,32] (i.e., the prediction scores of two semantically similar tags should be similar in the same image), image-level smoothness [13,30,32,20] (i.e., visually similar images have similar tags), low rank assumption [2] (i.e., the whole tag space is spanned by a lower-dimensional space), and semantic hierarchy [30,25] (i.e. parent tags in a hierarchy are as probable as their children).…”

Socializing the Semantic Gap

“…In addition to tags and images, this group of works exploits user information, motivated from varied perspectives. User information ranges from the simplest user identities [Li et al 2009b], to tagging preferences [Sawant et al 2010], to user reliability [Ginsca et al 2014], to image group memberships [Johnson et al 2015]. With the hypothesis that a specific tag chosen by many users to label visually similar images is more likely to be relevant with respect to the visual content, Li et al [2009b] utilize user identity to ensure that learning examples come from distinct users.…”

“…In McParlane et al [2013b], time-constrained tag co-occurrence statistics are considered to refine the output of visual classifiers for tag assignment. In their follow-up work [McParlane et al 2013a], location-constrained tag co-occurrence computed from images Johnson et al [2015], social network metadata such as image groups membership or contacts of users is employed to resolve ambiguity in visual appearance. Comparing the three groups, tag + image appears to be the mainstream, as evidenced by the imbalanced distribution in Table I.…”

Algorithmic Research for Image Annotation Based on Region Convolution Neural Network

“…In the field of image processing and pattern recognition, the research effort has transformed from image recognition to image annotation step by step [1]. According to machine learning and pattern recognition algorithm, the traditional image recognition can finish the work of feature extraction and classification.…”