Parallel field alignment for cross media retrieval

Abstract

Cross media retrieval systems have received increasing interest in recent years. Due to the semantic gap between lowlevel features and high-level semantic concepts of multimedia data, many researchers have explored joint-model techniques in cross media retrieval systems. Previous joint-model approaches usually focus on two traditional ways to design cross media retrieval systems: (a) fusing features from different media data; (b) learning different models for different media data and fusing their outputs. However, the process of fusing features or outputs will lose both low-and highlevel abstraction information of media data. Hence, both ways do not really reveal the semantic correlations among the heterogeneous multimedia data. In this paper, we introduce a novel method for the cross media retrieval task, named Parallel Field Alignment Retrieval (PFAR), which integrates a manifold alignment framework from the perspective of vector fields. Instead of fusing original features or outputs, we consider the cross media retrieval as a manifold alignment problem using parallel fields. The proposed manifold alignment algorithm can effectively preserve the metric of data manifolds, model heterogeneous media data and project their relationship into intermediate latent semantic spaces during the process of manifold alignment. After the alignment, the semantic correlations are also determined. In this way, the cross media retrieval task can be resolved by the determined semantic correlations. Comprehensive experimental results have demonstrated the effectiveness of our approach.

Hetero-Manifold Regularisation for Cross-Modal Hashing

“…The methods, such as [2], [5], [15], [19], support our view that exploiting the manifold structure is very important for boosting the performance of cross-model retrieval. However, no general frameworks for multi-modalities are available, no higher-order relationships have been considered, and, except for CHMIS [5], most existing methods can hardly be extended to more complex multi-modalities.…”

“…For example, in [1], a non-linear dimension reduction technique is introduced for cross-modal retrieval, where bimodal data are represented in a common lowdimensional Euclidean space and the cross-modal similarity is defined by using the Euclidean distance in the learned space. Mao et al [2] propose a cross-modal retrieval algorithm based on parallel field alignment in which heterogeneous data are mapped into a common Euclidean space to measure the similarity between heterogeneous data. Deep learning [9], [14], [16] is also employed to learn a common feature space which could be shared by heterogeneous data.…”

“…Firstly, multi-modal data are treated as an ensemble of homogeneous data, which are modeled as multiple homogeneous manifolds, such as [2], [5], [6], [33]. For example, Gao et al [33] constructed a similarity graph matrix for each uni-modal feature or label feature, and then learned an optimal similarity graph matrix for the given multi-modal data by fusing the similarity information of uni-modal similarity graph matrices and the label information with semi-supervised learning.…”

“…Some methods focus on the supervision information including correspondences [1], semantic correlation [2], pairwise sets [3] and semantic affinities [4] between heterogeneous data, while others including composite multiple information sources [5], α-average technique [6], [7], Markov random field [8] and deep neural networks [9] emphasise the value of homogeneous manifold in the problem of multi-modal similarity learning in a common space.…”

“…are limited to two modalities [2], [9], [10], [11], [12], [13] or strive to cope with more than two modalities but are still evaluated on the datasets with only two modalities [5], [14], [15].…”