Multimodal fusion using learned text concepts for image categorization

Abstract

Conventional image categorization techniques primarily rely on low-level visual cues. In this paper, we describe a multimodal fusion scheme which improves the image classification accuracy by incorporating the information derived from the embedded texts detected in the image under classification. Specific to each image category, a text concept is first learned from a set of labeled texts in images of the target category using Multiple Instance Learning [1]. For an image under classification which contains multiple detected text lines, we calculate a weighted Euclidian distance between each text line and the learned text concept of the target category. Subsequently, the minimum distance, along with lowlevel visual cues, are jointly used as the features for SVM-based classification. Experiments on a challenging image database demonstrate that the proposed fusion framework achieves a higher accuracy than the state-of-art methods for image classification.

Multimodal fusion for multimedia analysis: a survey

“…In the area of image classification, Zhu et al [156] have reported a multimodal fusion framework to classify the images that have embedded text within their spatial coordinates. The fusion process followed two steps.…”

“…Bayesian inference method has been successfully used to fuse multimodal information (at the feature level and at the decision level) for performing various multimedia Multimodal fusion using visual and text cues for image classification based on pair-wise SVM classifier [156] analysis tasks. An example of Bayesian inference fusion at the feature level is the work by Pitsikalis et al [102] for audio-visual speech recognition.…”

“…Similarly, for the video shot retrieval some researchers use mean average precision [83]. While performing image categorization, the accuracy of the classification is measured in terms of image category detection rate [156].…”

Words Matter: Scene Text for Image Classification and Retrieval

“…To classify images, Wang et al [54] propose to combine visual cues with corresponding user tags. In [66], the authors exploit textual information by extracting visual features around the text regions and combining them with global visual features. Different from these works, in this paper, textual information is obtained from detected and recognized text in natural scene images.…”

Video Analysis and Indexing

“…The increasing availability of online digital video has rekindled interest in the problems of how to index multimedia information sources automatically and how to browse and manipulate them efficiently (David, 1998) (Snoek & Worring, 2005) (Zhu et al, 2006). The need for efficient content-based video indexing and retrieval has increased due to the rapid growth of video data available to consumers.…”