Integration of spatial relationships in visual language model for scene retrieval

Abstract

In this paper, we describe a method to use a graph-based language modeling approach for image retrieval and image categorization. We first mapped image regions to induced concepts and then spatial relationships between these regions to build a graph representation of images. Our method allows to deal with different scenarii, where isolated images or groups of images are used for training and testing. The results obtained on an image categorization problem comprising of 3849 images from 101 landmarks of Singapore show that (a) the procedure to automatically induce concepts from an image is effective, and (b) the use of spatial relationships, in addition to concepts, for representing an image content helps improve the classifier accuracy.This approach is the first one, to our knowledge, to present a complete extension of the language modeling approach from information retrieval to the problem of graph-based image categorization and retrieval.

Visual graph modeling for scene recognition and mobile robot localization

“…The third step is related to the fact that we want to retrieve relevant images to a given query. Therefore, we extend the work in [21,22]bytaking into account the different types of image representations and spatial relations during matching by computing likelihood of two graphs using a language model framework.…”

“…Again, the quantities #(c, c ′ , l, D) and #(c, c ′ , * , D) are counted in a similar way but computed on the whole collection D (i.e., over the union of all the graphs from all the documents in the collection). This graph model is a generalization of the model defined in [21] which corresponds to the special case where only one concept set and one relation set are used. In some special cases, our model corresponds to the standard language model (LM) used in [15,20] where relations are not considered (i.e., documents and queries correspond to multiple bag-of-words model) .…”

“…Such language models are well-defined theoretically, and also have shown interesting experimental results as synthesized in [16]. Therefore, our main focus in this paper is to propose an extension of language models [21] in the context of graph-based representation for image content.…”