CrossATNet - a novel cross-attention based framework for sketch-based image retrieval

Abstract

We propose a novel framework for cross-modal zero-shot learning (ZSL) in the context of sketch-based image retrieval (SBIR). Conventionally, the SBIR schema mainly considers simultaneous mappings among the two image views and the semantic side information. Therefore, it is desirable to consider fine-grained classes mainly in the sketch domain using highly discriminative and semantically rich feature space. However, the existing deep generative modelling based SBIR approaches majorly focus on bridging the gaps between the seen and unseen classes by generating pseudo-unseen-class samples. Besides, violating the ZSL protocol by not utilizing any unseen-class information during training, such techniques do not pay explicit attention to modelling the discriminative nature of the shared space. Also, we note that learning a unified feature space for both the multi-view visual data is a tedious task considering the significant domain difference between sketches and the colour images. In this respect, as a remedy, we introduce a novel framework for zero-shot SBIR. While we define a cross-modal triplet loss to ensure the discriminative nature of the shared space, an innovative cross-modal attention learning strategy is also proposed to guide feature extraction from the image domain exploiting information from the respective sketch counterpart. In order to preserve the semantic consistency of the shared space, we consider a graph CNN based module which propagates the semantic class topology to the shared space. To ensure an improved response time during inference, we further explore the possibility of representing the shared space in terms of hash-codes. Experimental results obtained on the benchmark TU-Berlin and the Sketchy datasets confirm the superiority of CrossATNet in yielding the state-of-the-art results.

Interband Retrieval and Classification Using the Multilabeled Sentinel-2 BigEarthNet Archive

“…These cross-sensor retrieval techniques also extend to retrieving unseen class images upon deployment, commonly called zero-shot cross-modal retrieval [45]. As one of the major bottlenecks of solving remote sensing problems using deep learning is the lack of annotated samples for training, these cross-sensor zero-shot retrieval has received a lot of attention recently [18].…”

“…Recently, we have seen a lot of focus on cross-sensor/cross-modal retrieval techniques in RS using various learning techniques [15], [16]. Some of the notable works in this domain are presented in [17], [18], [19], [20]. Several literary works in this domain have tried to exploit the conventional triplet loss or the Siamese loss function to discriminate classes within a fine-grained dataset.…”

Discriminative Similarity-Balanced Online Hashing for Supervised Image Retrieval

“…Most existing online hashing methods have been devoted to the trade-off between accuracy and efficiency [15][16][17]. According to the learning strategy, people divide these techniques into unsupervised online hashing and supervised online hashing [18][19][20]. e well-known unsupervised methods mainly include online sketch hashing (SketchHash) [21], FasteR online sketch hash (FROSH) [22], and zero-mean sketch [23].…”

Zero-Shot Sketch Based Image Retrieval using Graph Transformer

“…However, while [9], [10], [19] used the semantic space, they made the semantic space latent and learnable causing the network to eventually loose the classwise topology information as the network is trained for more number of epochs. To preserve the original topology of the semantic space, [21], [22] proposed using a graph convolution network (GCN) [23]. While in [22] the authors use a GCN directly on the semantic graph, in [21] the authors create a fully-connected graph whose edge weights correspond to the semantic distances and the node features comprise of classwise visual features.…”

“…To preserve the original topology of the semantic space, [21], [22] proposed using a graph convolution network (GCN) [23]. While in [22] the authors use a GCN directly on the semantic graph, in [21] the authors create a fully-connected graph whose edge weights correspond to the semantic distances and the node features comprise of classwise visual features.…”