CAESAR: concept augmentation based semantic representation for cross-modal retrieval

Proceedings of the 2022 International Conference on Multimedia Retrieval

Cai

Song

et al. 2022

Self Cite

Cross-modal hashing is a hot issue in the multimedia community, which is to generate compact hash code from multimedia content for efficient cross-modal search. Two challenges, i.e., (1) How to efficiently enhance cross-modal semantic mining is essential for cross-modal hash code learning, and (2) How to combine multiple semantic correlations learning to improve the semantic similarity preserving, cannot be ignored. To this end, this paper proposed a novel end-to-end cross-modal hashing approach, named Multiple Semantic Structure-Preserving Quantization (MSSPQ) that is to integrate deep hashing model with multiple semantic correlation learning to boost hash learning performance. The multiple semantic correlation learning consists of inter-modal and intra-modal pairwise correlation learning and Cosine correlation learning, which can comprehensively capture cross-modal consistent semantics and realize semantic similarity preserving. Extensive experiments are conducted on three multimedia datasets, which confirms that the proposed method outperforms the baselines. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval.

Section: Related Workmentioning

confidence: 99%

MSSPQ: Multiple Semantic Structure-Preserving Quantization for Cross-Modal Retrieval

Proceedings of the 2022 International Conference on Multimedia Retrieval

Cai

Song

et al. 2022

Self Cite

“…Compared with these traditional techniques, deep neural networks, such as CNN [10], RNN, etc., have stronger semantic representation capability and larger parameter capacity, which are able to capture more high-level semantics from multi-modal content. Other advanced techniques, such as attention mechanism and semantic augmentation [11], are used to realize cross-modal semantic alignment.…”

Section: Introductionmentioning

confidence: 99%

Multi-Graph Based Hierarchical Semantic Fusion for Cross-Modal Representation

2021 IEEE International Conference on Multimedia and Expo (ICME)

Zhang

Song

et al. 2021

Self Cite

The main challenge of cross-modal retrieval is how to efficiently realize semantic alignment and reduce the heterogeneity gap. However, existing approaches ignore the multigrained semantic knowledge learning from different modalities. To this end, this paper proposes a novel end-to-end cross-modal representation method, termed as Multi-Graph based Hierarchical Semantic Fusion (MG-HSF). This method is an integration of multi-graph hierarchical semantic fusion with cross-modal adversarial learning, which captures finegrained and coarse-grained semantic knowledge from crossmodal samples, and generate modalities-invariant representations in a common subspace. To evaluate the performance, extensive experiments are conducted on three benchmarks. The experimental results show that our method is superior than the state-of-the-arts.

“…As stated in existing literatures [4,28,35,40], the major roadblocks of cross-modal retrieval are two folds, namely (1) cross-modal heterogeneity that hinders the similarity measurement between samples from different modalities, and (2) semantic gap existing between multi-modal data and human understanding, which obstructs the cross-modal semantic alignment. According to the type of representation, cross-modal retrieval techniques can be grouped into two categories, i.e., the real value-based methods [38,48] and binary value-based (hashing) approaches [15,18,43]. As the higher efficiency of retrieval and lower cost of storage, more attention has been paid to cross-modal hashing for big data applications.…”

Section: Introductionmentioning

confidence: 99%

M2guda

Zhang

Zhi

Proceedings of the 2021 International Conference on Multimedia Retrieval

et al. 2021

Self Cite

Cross-modal hashing is a critical but very challenging task that is to retrieve similar samples of one modality via queries of other modalities. To improve the unsupervised cross-modal hashing, domain adaptation techniques can be used to support unsupervised hashing learning by transferring semantic knowledge from labeled source domain to unlabeled target domain. However, there are two problems that cannot be ignored: (1) most of domain adaptation based researches mainly focused on unimodal hashing or cross-modal real value-based retrieval but the study for cross-modal hashing is limited; (2) most existing studies only consider one or two consistency constraints during the domain adaptation learning. To this end, this paper propose a novel end-to-end framework to realize unsupervised domain adaptation for cross-modal hashing. This method, dubbed M 2 GUDA, including four different consistency constraints: structure consistency, domain consistency, semantic consistency and modality consistency for domain adaptation learning. Besides, to enhance the structure consistency learning, we develop a multimetrics graph modeling method to capture structure information comprehensively. Extensive experiments are performed on three common used benchmarks to evaluate the effectivity of our method. The results show that our method outperforms several state-of-theart cross-modal hashing methods. CCS CONCEPTS• Information systems → Multimedia and multimodal retrieval.