Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Luo, Kaiyi; Zhang, Chao; Li, Huaxiong; Jia, Xiuyi; Chen, Chunlin

doi:10.1109/tmm.2023.3245400

Cited by 22 publications

(4 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering large-scale retrieval applications and unequal hash length encoding scenarios, discrete asymmetric hashing (DAH) (Zhang et al 2023b) proposes a flexible framework. To enhance the discrimination of hash codes, adaptive marginalized semantic hashing (AMSH) (Luo et al 2023) proposes the adaptive margin matrices to alleviate the rigid zero-one linear regression. However, these methods unconsciously ignore the influence of noise and outliers in the hashing learning process.…”

Section: Related Workmentioning

confidence: 99%

“…To verify the performance of our method, we compare DSCMH with thirteen state-of-the-art cross-modal hashing methods, including RFDH (Wang, Wang, and Gao 2017), LCMFH (Wang et al 2018), DLFH (Jiang and Li 2019), MTFH (Liu et al 2019), FCMH (Wang et al 2021b), FDDH (Liu, Wang, and Cheung 2021), BATCH (Wang et al 2021c), EDMH (Chen et al 2022), DAH (Zhang et al 2023b), ALECH (Li et al 2023), WASH (Zhang et al 2023a), and AMSH (Luo et al 2023).…”

Section: Baselines and Implementationmentioning

confidence: 99%

See 1 more Smart Citation

Dual Self-Paced Cross-Modal Hashing

Sun,

Dai,

Ren

et al. 2024

AAAI

View full text Add to dashboard Cite

Cross-modal hashing~(CMH) is an efficient technique to retrieve relevant data across different modalities, such as images, texts, and videos, which has attracted more and more attention due to its low storage cost and fast query speed. Although existing CMH methods achieve remarkable processes, almost all of them treat all samples of varying difficulty levels without discrimination, thus leaving them vulnerable to noise or outliers. Based on this observation, we reveal and study dual difficulty levels implied in cross-modal hashing learning, \ie instance-level and feature-level difficulty. To address this problem, we propose a novel Dual Self-Paced Cross-Modal Hashing (DSCMH) that mimics human cognitive learning to learn hashing from ``easy'' to ``hard'' in both instance and feature levels, thereby embracing robustness against noise/outliers. Specifically, our DSCMH assigns weights to each instance and feature to measure their difficulty or reliability, and then uses these weights to automatically filter out the noisy and irrelevant data points in the original space. By gradually increasing the weights during training, our method can focus on more instances and features from ``easy'' to ``hard'' in training, thus mitigating the adverse effects of noise or outliers. Extensive experiments are conducted on three widely-used benchmark datasets to demonstrate the effectiveness and robustness of the proposed DSCMH over 12 state-of-the-art CMH methods.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Baselines and Implementationmentioning

confidence: 99%

Dual Self-Paced Cross-Modal Hashing

Sun,

Dai,

Ren

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…1) Incomplete Cross-modal Retrieval: Efforts in addressing the challenge of imbalanced or incomplete cross-modal retrieval focus on mitigating the issue of unbalanced or unpaired multi-modal samples in training data through the utilization of either shallow or deep learning techniques. In the shallow learning methods [166]- [168], typically, Robust Unsupervised Cross-Modal Hashing (RUCMH) [167] maps multiple heterogeneous spaces to a common semantic space for object reconstruction, without relying on paired data. Advancements in deep learning lead to Triplet Fusion Network Hashing (TFNH) [220], which utilizes a triplet network with zero padding and data classifiers to ensure effective learning for unpaired data.…”

Section: E Cross-modal Retrieval Under Special Retrieval Scenariomentioning

confidence: 99%

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

Zhu

Wang

et al. 2023

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

Deep cross-modal hashing retrieval models inherit the vulnerability of deep neural networks. They are vulnerable to adversarial attacks, especially for the form of subtle perturbations to the inputs. Although many adversarial attack methods have been proposed to handle the robustness of hashing retrieval models, they still suffer from two problems: 1) Most of them are based on the white-box settings, which is usually unrealistic in practical application. 2) Iterative optimization for the generation of adversarial examples in them results in heavy computation. To address these problems, we propose an Efficient Query-based Black-Box Attack (EQB 2 A) against deep cross-modal hashing retrieval, which can efficiently generate adversarial examples for the black-box attack. Specifically, by sending a few query requests to the attacked retrieval system, the cross-modal retrieval model stealing is performed based on the neighbor relationship between the retrieved results and the query, thus obtaining the knockoffs to substitute the attacked system. A multi-modal knockoffs-driven adversarial generation is proposed to achieve efficient adversarial example generation. While the entire network training converges, EQB 2 A can efficiently generate adversarial examples by forward-propagation with only given benign images. Experiments show that EQB 2 A achieves superior attacking performance under the black-box setting.

show abstract

“…Representation learning can effectively deal with this problem. In such methods, the aim is to learn a function that can transform different modalities into a common feature space [4,5], where we can compare them directly. Due to the quick expansion of the data scale and the decline of data retrieval efficiency, the hashing codes are applied to cross-modal retrieval tasks [6][7][8].…”

Section: Introductionmentioning

confidence: 99%

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Ren,

2024

Front. Phys.

View full text Add to dashboard Cite

Along with the continuous breakthrough and popularization of information network technology, multi-modal data, including texts, images, videos, and audio, is growing rapidly. We can retrieve different modal data to meet our needs, so cross-modal retrieval has important theoretical significance and application value. In addition, because the data of different modalities can be mutually retrieved by mapping them to a unified Hamming space, hash codes have been extensively used in the cross-modal retrieval field. However, existing cross-modal hashing models generate hash codes based on single-dimension data features, ignoring the semantic correlation between data features in different dimensions. Therefore, an innovative cross-modal retrieval method using Multi-Dimensional Feature Fusion Hashing (MDFFH) is proposed. To better get the image’s multi-dimensional semantic features, a convolutional neural network, and Vision Transformer are combined to construct an image multi-dimensional fusion module. Similarly, we apply the multi-dimensional text fusion module to the text modality to obtain the text’s multi-dimensional semantic features. These two modules can effectively integrate the semantic features of data in different dimensions through feature fusion, making the generated hash code more representative and semantic. Extensive experiments and corresponding analysis results on two datasets indicate that MDFFH’s performance outdoes other baseline models.

show abstract

Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Cited by 22 publications

References 51 publications

Dual Self-Paced Cross-Modal Hashing

Dual Self-Paced Cross-Modal Hashing

Efficient Query-based Black-box Attack against Cross-modal Hashing Retrieval

Cross-modal retrieval based on multi-dimensional feature fusion hashing

Contact Info

Product

Resources

About