Global semantics correlation transmitting and learning for sketch-based cross-domain visual retrieval

Abstract

Sketch-based cross-domain visual data retrieval is the process of searching for images or 3D models using sketches as input. Achieving feature alignment is a significantly challenging task due to the high heterogeneity of cross-domain data. However, the alignment process faces significant challenges, such as domain gap, semantic gap, and knowledge gap. The existing methods adopt different ideas for sketch-based image and 3D shape retrieval tasks, one is domain alignment, and the other is semantic alignment. Technically, both tasks verify the accuracy of extracted features. Hence, we propose a method based on the global feature correlation and the feature similarity for multiple sketch-based cross-domain retrieval tasks. Specifically, the data from various modalities are fed into separate feature extractors to generate original features. Then, these features are projected to the shared subspace. Finally, domain consistency learning, semantic consistency learning, feature correlation learning and feature similarity learning are performed jointly to make the projected features modality-invariance. We evaluate our method on multiple benchmark datasets. Where the MAP in Sketchy, TU-Berlin, SHREC 2013 and SHREC 2014 are 0.466, 0.473, 0.860 and 0.816. The extensive experimental results demonstrate the superiority and generalization of the proposed method, compared to the state-of-the-art approaches. The in-depth analyses of various design choices are also provided to gain insight into the effectiveness of the proposed method. The outcomes of this research contribute to advancing the field of sketch-based cross-domain visual data retrieval and are expected to be applied to a variety of applications that require efficient retrieval of cross-domain domain data.