Improvement of deep cross-modal retrieval by generating real-valued representation

Abstract

The cross-modal retrieval (CMR) has attracted much attention in the research community due to flexible and comprehensive retrieval. The core challenge in CMR is the heterogeneity gap, which is generated due to different statistical properties of multi-modal data. The most common solution to bridge the heterogeneity gap is representation learning, which generates a common sub-space. In this work, we propose a framework called “Improvement of Deep Cross-Modal Retrieval (IDCMR)”, which generates real-valued representation. The IDCMR preserves both intra-modal and inter-modal similarity. The intra-modal similarity is preserved by selecting an appropriate training model for text and image modality. The inter-modal similarity is preserved by reducing modality-invariance loss. The mean average precision (mAP) is used as a performance measure in the CMR system. Extensive experiments are performed, and results show that IDCMR outperforms over state-of-the-art methods by a margin 4% and 2% relatively with mAP in the text to image and image to text retrieval tasks on MSCOCO and Xmedia dataset respectively.

Impact of Binary-Valued Representation on the Performance of Cross-Modal Retrieval System

“…The CMR approaches based on pairwise labels preserve similarity between corresponding data points. Deep pairwise-supervised hashing (DPSH) (Li et al, 2015), (Bhatt and Ganatra, 2021). All above approaches preserve similarity between corresponding data points but fail to preserve relative similarity between heterogeneous modalities.…”