Multimodal indexing based on semantic cohesion for image retrieval

Abstract

This paper introduces two novel strategies for representing multimodal images with application to multimedia image retrieval. We consider images that are composed of both text and labels: while text describes the image content at a very high semantic level (e.g., making reference to places, dates or events), labels provide a mid-level description of the image (i.e., in terms of the objects that can be seen in the image). Accordingly, the main assumption of this work is that by combining information from text and labels we can develop very effective retrieval methods. We study standard information fusion techniques for combining both sources of information. However, whereas the performance of such techniques is highly competitive, they cannot capture effectively the content of images. Therefore, we propose two novel representations for multimodal images that attempt to exploit the semantic cohesion among terms from different modalities. Such representations are based on distributional term representations widely used in computational linguistics. Under the considered representations the content of an image is modeled by a distribution of co-occurrences over terms or of occurrences over other images, in such a way that the representation can be considered an expansion of the multimodal terms in the image. We report experimental results using the SAIAPR TC12 benchmark on two sets of topics used in ImageCLEF competitions with manually and automatically generated labels. Experimental results show that the proposed representations outperform significantly both, standard multimodal techniques and unimodal methods. Results on manually assigned labels provide an upper bound in the retrieval performance that can be obtained, whereas results with automatically generated labels are encouraging. The novel representations are able to capture more effectively the content of multimodal images. We emphasize that although we have applied our representations to multimedia image retrieval the same formulation can be adopted for modeling other multimodal documents (e.g., videos).

Distributional Term Representations for Short-Text Categorization

“…On the other hand, DTRs have been mainly used in term classification and term clustering tasks [11], also they have been recently used in multimedia image retrieval [3]. DTRs, however, have not been used for short-text categorization, despite their potential benefits for document expansion.…”

An energy-based model for region-labeling

“…where #(a j ) is the number of images in the training set in which label a j occurs and #ða j 1 ; a j 2 Þ is the number of images in the training set in which both a j 1 and a j 2 co-occur 3 . For specific labels a j 1 and a j 2 , the higher A(j 1 ,j 2 ) the more both labels are related.…”

“…However, in this work we do not considered probabilities and instead used (un-normalized) raw counts of word co-occurrence, thus the ratio #ðaj 1 ;aj 2 Þ #ðj 1 Þ#ðj 2 Þ is just an indicator of association between labels. 3 In previous work we have investigated the use of different corpora for computing…”

“…Labels generated with RL methods can also be helpful to improve the performance of text-based image retrieval (TBIR, i.e., image search through text that has been manually assigned to images) methods, in collections where images are accompanied by textual descriptors [3]. This is due to the fact that labels used in AIA correspond to visual information in the image, while manually assigned annotations often refer to high-level semantic information of the image, see Fig.…”

“…This is due to the fact that labels used in AIA correspond to visual information in the image, while manually assigned annotations often refer to high-level semantic information of the image, see Fig. 1; in consequence, both text and labels can be considered complimentary [3]. Thus, RL has an important impact into the multimedia image retrieval domain and therefore the development of effective RL techniques is a crucial task.…”