UCMH: Unpaired cross-modal hashing with matrix factorization

Gao, Jing; Zhang, Wenjun; Zhong, Fangming; Chen, Zhikui

doi:10.1016/j.neucom.2020.08.029

Cited by 13 publications

(10 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To evaluate the performance of the proposed FMBSD method, we compare its experimental results with the results of several state-of-the-art methods of different types developed for realistic problems in practical scenario, containing WMCA (Lampert and Krömer, 2010), MMPDL (Liu et al, 2018), FlexCMH (Yu et al, 2020), PMH , GSPH (Mandal et al, 2019) and UCMH (Gao et al, 2020a), which are without pointwise correspondences (unpaired methods). The first two methods use latent subspace approaches across multimodal data, while the next methods are hashing-based ones.…”

Section: Comparison Methodsmentioning

confidence: 99%

“…These two types of correspondences, pointwise and batch correspondences, are very strong assumptions and cannot handle in many of realistic problems. In last recent years, a few works are exploited for the practical problems, such as UCMH (Gao et al, 2020a) which address the data with completely unpaired relationships by mapping data of different modalities to their respective semantic spaces, and FlexCMH (Yu et al, 2020) which introduces a clustering-based matching strategy to find the potential correspondences between samples across modalities.…”

Section: Multimodal Multiclass Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Behmanesh,

Adibi,

Chanussot

et al. 2021

Preprint

View full text Add to dashboard Cite

Multimodal manifold modeling methods extend the spectral geometry-aware data analysis to learning from several related and complementary modalities. Most of these methods work based on two major assumptions: 1) there are the same number of homogeneous data samples in each modality, and 2) at least partial correspondences between modalities are given in advance as prior knowledge. This work proposes two new multimodal modeling methods. The first method establishes a general analyzing framework to deal with the multimodal information problem for heterogeneous data without any specific prior knowledge. For this purpose, first, we identify the localities of each manifold by extracting local descriptors via spectral graph wavelet signatures (SGWS). Then, we propose a manifold regularization framework based on the functional mapping between SGWS descriptors (FMBSD) for finding the pointwise correspondences. The second method is a manifold regularized multimodal classification based on pointwise correspondences (M 2 CPC) used for the problem of multiclass classification of multimodal heterogeneous, which the correspondences between modalities are determined based on the FMBSD method. The experimental results of evaluating the FMBSD method on three common cross-modal retrieval datasets and evaluating the (M 2 CPC) method on three benchmark multimodal multiclass classification datasets indicate their effectiveness and superiority over state-of-the-art methods.

show abstract

Section: Comparison Methodsmentioning

confidence: 99%

Section: Multimodal Multiclass Classificationmentioning

confidence: 99%

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Behmanesh,

Adibi,

Chanussot

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…methods are not able to solve the UCMR problem due to their reliance on paired data. To the best of our knowledge, there are few literatures dedicated to the UCMR problem [29], [30], [31], [32]. (2) Some supervised CMH methods train a classifier to preserve high-level semantic information based on the original logical label matrix, which ignores the distance between different classes.…”

Section: Labelsmentioning

confidence: 99%

“…Generalized Semantic Preserving Hashing (GSPH) [29] seeks to factorize the similarity matrix into the dot product of two binary matrices, thus embedding semantic information into the to-be-learned binary codes, and applies kernel logistic regression to obtain modality-specific hash functions. Unpaired Cross-Modal Hashing (UCMH) [30] utilizes matrix factorization to generate modality-specific latent representations. To preserve semantic similarities, UCMH constructs the Laplacian matrices with labels, which is time-consuming.…”

Section: B Unpaired Cross-modal Hashingmentioning

confidence: 99%

“…These baselines are implemented with the codes provided by the authors, and the parameters of these baselines are carefully tuned according to the original paper to reach their best performance. For the unpaired scenario, SAPMH [27], GSPH [29], UCMH [30], RUCMH [31], EDMH [44] and FMH [32] are adopted for comparison. Among these baselines, RUCMH is an unsupervised method and the others are supervised ones.…”

Section: B Baselines and Implementationmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Luo¹,

Zhang²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

In recent years, Cross-Modal Hashing (CMH) has aroused much attention due to its fast query speed and efficient storage. Previous literatures have achieved promising results for Cross-Modal Retrieval (CMR) by discovering discriminative hash codes and modality-specific hash functions. Nonetheless, most existing CMR works are subjected to some restrictions: 1) It is assumed that data of different modalities are fully paired, which is impractical in real applications due to sample missing and false data alignment, and 2) binary regression targets including the label matrix and binary codes are too rigid to effectively learn semantic-preserving hash codes and hash functions. To address these problems, this paper proposes an Adaptive Marginalized Semantic Hashing (AMSH) method which not only enhances the discrimination of latent representations and hash codes by adaptive margins, but also can be used for both paired and unpaired CMR. As a two-step method, in the first step, AMSH generates semantic-aware modality-specific latent representations with adaptively marginalized labels, which enlarges the distances between different classes, and exploits the labels to preserve the inter-modal and intra-modal semantic similarities into latent representations and hash codes. In the second step, adaptive margin matrices are embedded into the hash codes, and enlarge the gaps between positive and negative bits, which improves the discrimination and robustness of hash functions. On this basis, AMSH generates similarity-preserving hash codes and robust hash functions without strict one-to-one data correspondence requirement. Experiments are conducted on several benchmark datasets to demonstrate the superiority and flexibility of AMSH over some state-of-the-art CMR methods.

show abstract

Unpaired robust hashing with noisy labels for zero-shot cross-modal retrieval

Yong,

Shu,

2024

Engineering Applications of Artificial Intelligence

View full text Add to dashboard Cite

UCMH: Unpaired cross-modal hashing with matrix factorization

Cited by 13 publications

References 19 publications

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Adaptive Marginalized Semantic Hashing for Unpaired Cross-Modal Retrieval

Unpaired robust hashing with noisy labels for zero-shot cross-modal retrieval

Contact Info

Product

Resources

About