Semantic Concept Co-Occurrence Patterns for Image Annotation and Retrieval

Abstract

Describing visual image contents by semantic concepts is an effective and straightforward way to facilitate various high level applications. Inferring semantic concepts from low-level pictorial feature analysis is challenging due to the semantic gap problem, while manually labeling concepts is unwise because of a large number of images in both online and offline collections. In this paper, we present a novel approach to automatically generate intermediate image descriptors by exploiting concept co-occurrence patterns in the pre-labeled training set that renders it possible to depict complex scene images semantically. Our work is motivated by the fact that multiple concepts that frequently co-occur across images form patterns which could provide contextual cues for individual concept inference. We discover the co-occurrence patterns as hierarchical communities by graph modularity maximization in a network with nodes and edges representing concepts and co-occurrence relationships separately. A random walk process working on the inferred concept probabilities with the discovered co-occurrence patterns is applied to acquire the refined concept signature representation. Through experiments in automatic image annotation and semantic image retrieval on several challenging datasets, we demonstrate the effectiveness of the proposed concept co-occurrence patterns as well as the concept signature representation in comparison with state-of-the-art approaches.

Semantic image annotation using convolutional neural network and wordnet ontology

“…We can also find the semantic similarity between the predicted class labels. As there is high probability for semantically similar concepts to co-occur in an image [13], for example 'cars' and 'trucks'. The semantic similarity score can help in accurately labelling images.…”

“…The proposed model using discrete cosine transform (DCT) for feature extraction. In 2016, Linan Feng proposed a system for image retrieval and annotation based on semantic concept co-occurrence [13]. In his paper, he describes a new approach to automatically generate descriptions for the images by considering concept co-occurrence patterns in the pre-labelled training dataset that makes it possible to create complex semantic descriptions for scene images.…”

Automatic Image Annotation using Possibilistic Clustering Algorithm

“…Motivated by the fact that multiple concepts that frequently co-occur over images form patterns that could provide contextual cues for proper concept inference, the researchers in [28] outlined an AIA approach that exploited the co-occurrence patterns as hierarchical communities in the pre-labelled training set. In addition, the model called automatic linguistic indexing of pictures (ALIP) [29] was used to compute the likelihood of the occurrence of images based on a describing stochastic process.…”

Semantic concept based video retrieval using convolutional neural network

“…A good survey on concept-based video retrieval is presented by Snoek and Worring [2]. Feng and Bhanu [3] discussed the concept and use of concept co-occurrence patterns for image annotation and retrieval. Kuo et al [4] presented the work on the use of deep convolutional neural network for image retrieval.…”