Multimedia retrieval based on non-linear graph-based fusion and partial least squares regression

Abstract

Heterogeneous sources of information, such as images, videos, text and metadata are often used to describe different or complementary views of the same multimedia object, especially in the online news domain and in large annotated image collections. The retrieval of multimedia objects, given a multimodal query, requires the combination of several sources of information in an efficient and scalable way. Towards this direction, we provide a novel unsupervised framework for multimodal fusion of visual and textual similarities, which are based on visual features, visual concepts and textual metadata, integrating non-linear graph-based fusion and Partial Least Squares Regression. The fusion strategy is based on the construction of a multimodal contextual similarity matrix and the non-linear combination of relevance scores from query-based similarity vectors. Our framework can employ more than two modalities and high-level information, without increase in memory complexity, when compared to state-of-the-art baseline methods. The experimental comparison is done in three public multimedia collections in the multimedia retrieval task. The results have shown that the proposed method outperforms the baseline methods, in terms of Mean Average Precision and Precision@20.

A Cross-Media Retrieval Method Based on Semisupervised Learning and Alternate Optimization

“…PLSR (Partial Least Squares Regression) [28] method is a regression modeling method of multidependent variable Y for multi-independent variable X. In the process of regression, the method considers both extracting the principal components in Y and X as much as possible (PCA-Principal Component Analysis) [29] and maximizing the correlation between the extracted principal components respectively (CCA).…”

A Review of Multisensor Data Fusion Solutions in Smart Manufacturing: Systems and Trends

“…In the work of [134], the multimodal fusion of visual and textual similarities was explored from the perspective of an unsupervised framework. These similarities are based on visual features and concepts, as well as textual metadata with the purpose of integrating nonlinear graph-based fusion and PLS regression.…”

Fusion of Compound Queries with Multiple Modalities for Known Item Video Search

“…A unifying model for unsupervised fusion of all similarities per modality has been presented in [10] and has been generalized to a non-linear fusion approach [2] that combines cross-media similarities with diffusion-based scores on the graph of items, in a non-linear but scalable way, for several modalities. Other methodologies for combining heterogeneous modalities involve Partial Least Squares [11], [12] and correlation matching, mapping multiple modalities to points in a common linear subspace. In [13] a video retrieval framework is proposed, which fuses textual and visual information in a non-linear way.…”

“…The values of α m , β m , γ m , m = 1, 2, 3 parameters are optimized following the methodology presented in [12], which involves keeping constant a set of parameters while examining the effect on the change of the others in the retrieval.…”