Self-Supervised Correlation Learning for Cross-Modal Retrieval

Liu, Yaxin; Wu, Jian; Qu, Leigang; Gan, Tian; Yin, Jianhua; Nie, Liqiang

doi:10.1109/tmm.2022.3152086

Cited by 25 publications

(9 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare our LDCA with more than 17 state-of-the-art methods in the experiments, namely MCCA, 29 DCCA, 30 JRL, 36 MvDA-VC, 48 ACMR, 7 CCL, 46 MCSM, 49 DSCMR, 1 SDML, 50 C-DCCA, 31 HCMSL, 32 DRSL, 6 BLN, 51 MARS, 2 URL, 52 SCL, 53 and AP-GRL 3 …”

Section: Methodsmentioning

confidence: 99%

Learning discriminative common alignments for cross-modal retrieval

Liu,

Chen,

Hong

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

Cross-modal retrieval aims to find alignment relationships between different modalities and then compute the semantic similarities used for ranking. Because of the data distribution difference and inherent heterogeneity gap between modalities, a classic solution is to learn common representations in the common space, which could preserve the discrimination among the samples from different categories and alleviate the cross-modal discrepancy. To achieve this, we propose a method, termed LDCA, to learn discriminative common alignments based on the modal representations. LDCA utilizes a modality invariance loss that pushes away the hardest negative sample to further reduce the cross-modal discrepancy at the feature level. In addition, LDCA seeks alignments in the label space to improve the intra-modal discrimination by an effective cross-modal label loss. Extensive experiments are conducted on five widely used cross-modal datasets to evaluate the proposed LDCA. The integral experimental results prove the method's superiority, and the comprehensive analyses verify the effectiveness of the method.

show abstract

Section: Methodsmentioning

confidence: 99%

Learning discriminative common alignments for cross-modal retrieval

Liu,

Chen,

Hong

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

show abstract

“…To validate the efficiency of our proposed FB-Net, we compare it with several state-of-the-art methods, in which seven non-DNN-based cross-modal retrieval methods (i.e., CCA [3], CMCP [23], JRL [25], JFSSL [26], and S 2 2UPG [27]) and nine DNN-based methods (i.e., DCCA [7], CCL [12], SCAN [15], GXN [28], VSESC [29], MAVA [30], SGRAF [31],SCL [41],CGMN [42],NAAF [32], and VSRN++ [33]) are contained. Note that the comparison methods are implemented using the authors' public source codes and are enumerated as follows.…”

Section: Compared Methodsmentioning

confidence: 99%

“…• SCL [41] takes advantage of the correlations between intra-and inter-modality items to acquire more discriminative features for multi-modal data.…”

Section: Compared Methodsmentioning

confidence: 99%

FB-Net: Dual-Branch Foreground-Background Fusion Network With Multi-Scale Semantic Scanning for Image-Text Retrieval

Liu

Pei

et al. 2023

IEEE Access

View full text Add to dashboard Cite

As a fundamental branch in cross-modal retrieval, image-text retrieval is still a challenging problem largely due to the complementary and imbalanced relationship between different modalities. However, existing works have not effectively scanned and aligned the semantic units distributed in different granularities of images and texts. To address these issues, we propose a dual-branch foreground-background fusion network (FB-Net), which is implemented by fully exploring and fusing the complementarity in semantic units collected from the foreground and background areas of instances (e.g., images and texts). Firstly, to generate multi-granularity semantic units from images and texts, multi-scale semantic scanning is conducted on both foreground and background areas through multi-level overlapped sliding windows. Secondly, to align semantic units between images and texts, the stacked cross-attention mechanism is used to calculate the initial image-text similarity. Thirdly, to further adaptively optimize the image-text similarity, the dynamically self-adaptive weighted loss is designed. Finally, to perform the image-text retrieval, the similarities between multi-granularity foreground and background semantic units are fused to obtain the final image-text similarity. Experimental results show that our proposed FB-Net outperforms representative state-of-the-art methods for image-text retrieval, and ablation studies further verify the effectiveness of each component in FB-Net.

show abstract

“…With the development of 3D sensors such as LiDAR, the acquisition of 3D data and the progress of 3D tasks become more active. In particular, great progress has been made in the 3D field based on point clouds (Wu et al 2022(Wu et al , 2023bHuang, Mei, and Zhang 2023;Liu et al 2023c). Yet, SOT remains challenging due to the variation in object appearance and the sparseness caused by sensors with inherent limitations.…”

Section: Introductionmentioning

confidence: 99%

M3SOT: Multi-Frame, Multi-Field, Multi-Space 3D Single Object Tracking

Liu,

Wu,

Gong

et al. 2024

AAAI

View full text Add to dashboard Cite

3D Single Object Tracking (SOT) stands a forefront task of computer vision, proving essential for applications like autonomous driving. Sparse and occluded data in scene point clouds introduce variations in the appearance of tracked objects, adding complexity to the task. In this research, we unveil M3SOT, a novel 3D SOT framework, which synergizes multiple input frames (template sets), multiple receptive fields (continuous contexts), and multiple solution spaces (distinct tasks) in ONE model. Remarkably, M3SOT pioneers in modeling temporality, contexts, and tasks directly from point clouds, revisiting a perspective on the key factors influencing SOT. To this end, we design a transformer-based network centered on point cloud targets in the search area, aggregating diverse contextual representations and propagating target cues by employing historical frames. As M3SOT spans varied processing perspectives, we've streamlined the network—trimming its depth and optimizing its structure—to ensure a lightweight and efficient deployment for SOT applications. We posit that, backed by practical construction, M3SOT sidesteps the need for complex frameworks and auxiliary components to deliver sterling results. Extensive experiments on benchmarks such as KITTI, nuScenes, and Waymo Open Dataset demonstrate that M3SOT achieves state-of-the-art performance at 38 FPS. Our code and models are available at https://github.com/ywu0912/TeamCode.git.

show abstract

Self-Supervised Correlation Learning for Cross-Modal Retrieval

Cited by 25 publications

References 38 publications

Learning discriminative common alignments for cross-modal retrieval

Learning discriminative common alignments for cross-modal retrieval

FB-Net: Dual-Branch Foreground-Background Fusion Network With Multi-Scale Semantic Scanning for Image-Text Retrieval

M3SOT: Multi-Frame, Multi-Field, Multi-Space 3D Single Object Tracking

Contact Info

Product

Resources

About