An information-theoretic framework for semantic-multimedia retrieval

Abstract

This article is set in the context of searching text and image repositories by keyword. We develop a unified probabilistic framework for text, image, and combined text and image retrieval that is based on the detection of keywords (concepts) using automated image annotation technology. Our framework is deeply rooted in information theory and lends itself to use with other media types. We estimate a statistical model in a multimodal feature space for each possible query keyword. The key element of our framework is to identify feature space transformations that make them comparable in complexity and density. We select the optimal multimodal feature space with a minimum description length criterion from a set of candidate feature spaces that are computed with the average-mutual-information criterion for the text part and hierarchical expectation maximization for the visual part of the data. We evaluate our approach in three retrieval experiments (only text retrieval, only image retrieval, and text combined with image retrieval), verify the framework's low computational complexity, and compare with existing state-of-the-art ad-hoc models.

Review of the field environmental sensing methods based on multi-sensor information fusion technology

“…Its application in the sensor data fusion process aims at drawing the conclusion of maximizing entropy. Wei et al [51] discussed the application of reliability theory of maximum entropy method in engineering, and studied and discussed the influence of sample mean and standard deviation on convergence, as well as the influence of sample size and maximum entropy method order on probability density function, transcendental probability, and reliability index accuracy. At present, entropy method has been applied to the integrated multimedia observation data classification by researchers.…”

Fusion of Global and Local Features with Multi-Inverted Indices for Efficient Image Retrieval

“…The goal of early fusion is to derive a common feature vector from all available features (e.g. [33], [34]). The advantage is the maximum amount of available information, including the correlation among features.…”

Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

“…The first strategy, called early fusion, performs fusion at the feature level (e.g. [18,4]), where features from the considered modalities are combined into a common feature vector. The second strategy, known as late fusion, fuses information at the decision level, meaning that each modality is first learned separately and the individual results are aggregated into a final common decision (e.g.…”