Comparative analysis on cross-modal information retrieval: A review

Kaur, Parminder; Pannu, Husanbir Singh; Malhi, Avleen

doi:10.1016/j.cosrev.2020.100336

Cited by 50 publications

(19 citation statements)

References 135 publications

(209 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As said in Section 1, there are two challenges in CS-CBRSIR, i.e., the heterogeneity gap issue [22] and the data shift problem [23]. Overall, the above two challenges can be regarded as the modality discrepancy problem.…”

Section: The Overview Of the Frameworkmentioning

confidence: 98%

“…Thus, cross-source CBRSIR (CS-CBRSIR) is proposed. CS-CBRSIR can be seen as a member of the cross-modal family [17][18][19][20][21], and it is confronted with the challenge of heterogeneity gaps [22] when measuring the resemblance between different types of HRRS data. Another challenge in CS-CBRSIR is the data shift problem [23] where the data distributions are different as the source and target images are acquired with various sensors.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

Shi

Tang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

Content-based remote sensing (RS) image retrieval (CBRSIR) is a critical way to organize high-resolution RS (HRRS) images in the current big data era. The increasing volume of HRRS images from different satellites and sensors leads to more attention to the cross-source CSRSIR (CS-CBRSIR) problem. Due to the data drift, one crucial problem in CS-CBRSIR is the modality discrepancy. Most existing methods focus on finding a common feature space for various HRRS images to address this issue. In this space, their similarity relations can be measured directly to obtain the cross-source retrieval results straight. This way is feasible and reasonable, however, the specific information corresponding to HRRS images from different sources is always ignored, limiting retrieval performance. To overcome this limitation, we develop a new model for CS-CBRSIR in this paper named dual modality collaborative learning (DMCL). To fully explore the specific information from diverse HRRS images, DMCL first introduces ResNet50 as the feature extractor. Then, a common space mutual learning module is developed to map the specific features into a common space. Here, the modality discrepancy is reduced from the aspects of features and their distributions. Finally, to supplement the specific knowledge to the common features, we develop modality transformation and the dual-modality feature learning modules. Their function is to transmit the specific knowledge from different sources mutually and fuse the specific and common features adaptively. The comprehensive experiments are conducted on a public dataset. Compared with many existing methods, the behavior of our DMCL is stronger. These encouraging results for a public dataset indicate that the proposed DMCL is useful in CS-CBRSIR tasks.

show abstract

Section: The Overview Of the Frameworkmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

Shi

Tang

et al. 2022

Remote Sensing

View full text Add to dashboard Cite

show abstract

“…The information overload triggered by the big data era has motivated researchers and practitioners to develop numerous automated information retrieval methods by using different yet often complementary approaches [25,18,32]. Such methods have been widely used in fields as digital libraries [16], information filtering and recommender systems [6,15], media search [26] and search engines [2].…”

Section: Related Workmentioning

confidence: 99%

From Scattered Sources to Comprehensive Technology Landscape: A Recommendation-based Retrieval Approach

Duong¹,

David²,

Dolamic³

et al. 2021

Preprint

View full text Add to dashboard Cite

Mapping the technology landscape is crucial for market actors to take informed investment decisions. However, given the large amount of data on the Web and its subsequent information overload, manually retrieving information is a seemingly ineffective and incomplete approach. In this work, we propose an end-to-end recommendation based retrieval approach to support automatic retrieval of technologies and their associated companies from raw Web data. This is a two-task setup involving (i) technology classification of entities extracted from company corpus, and (ii) technology and company retrieval based on classified technologies. Our proposed framework approaches the first task by leveraging DistilBERT which is a state-of-the-art language model. For the retrieval task, we introduce a recommendation-based retrieval technique to simultaneously support retrieving related companies, technologies related to a specific company and companies relevant to a technology. To evaluate these tasks, we also construct a data set that includes company documents and entities extracted from these documents together with company categories and technology labels. Experiments show that our approach is able to return 4 times more relevant companies while outperforming traditional retrieval baseline in retrieving technologies.

show abstract

“…Since the heterogeneous gap between modalities limits the similarly computation of parings, existing cross-modal learning methods [25,34,53] aim to learn a common space to generate modality gap-free representations so that eventually the transformed data can be compared using distance metrics, such as cosine similarity distance and Euclidean distance. Many models [4] have been proposed to learn such a shared space, and it is difficult to summarize these approaches with well-defined categories.…”

Section: Related Workmentioning

confidence: 99%

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Zeng¹,

Wu²,

Hattori³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning common subspace is prevalent way to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared when achieved in cross-modal retrieval. Previous cross-modal retrieval methods focus on projecting the data between modalities into a common latent subspace by learning the correlation between them to bridge the modality gap. However, due to the rich semantic information in the video, the heterogeneous nature of audio-visual data leads to heterogeneous gaps in the distribution of modalities, which are difficult to be bridged into a common subspace by the previous cross-modality learning methods, which may lead to the loss of important semantic content in the video when the modality gap is eliminated by supervised learning, e.g., optimizing pairwise correlation, while the semantics of the categories may undermine the properties of the original features In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

show abstract

Comparative analysis on cross-modal information retrieval: A review

Cited by 50 publications

References 135 publications

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

Dual Modality Collaborative Learning for Cross-Source Remote Sensing Retrieval

From Scattered Sources to Comprehensive Technology Landscape: A Recommendation-based Retrieval Approach

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Contact Info

Product

Resources

About