Fine-Grained Color Sketch-Based Image Retrieval

Abstract

We propose a novel fine-grained color sketch-based image retrieval (CSBIR) approach. The CSBIR problem is investigated for the first time using deep learning networks, in which deep features are used to represent color sketches and images. A novel ranking method considering both shape matching and color matching is also proposed. In addition, we build a CSBIR dataset with color sketches and images to train and test our method. The results show that our method has better retrieval performance.

Sketch-QNet: A Quadruplet ConvNet for Color Sketch-based Image Retrieval

“…Closer to our proposal is the work presented by Bui and Collomose [1], which processes shape and color independently using the well-known BoW strategy and features produced by gradient fields. Similar to this work is that proposed by Xia et al [17] which extracts shape features through a convolutional neural network to retrieve the N most similar images. The resulting ranking is then resorted using color histograms.…”

“…So far we know, the current methods dealing with color sketches for image retrieval are the approaches proposed by Bui and Collomosse [1] using gradient fields together with a BoW strategy; and the proposal of Xia et al [17] that uses a convnet to extract shape features. In general terms, both proposals are similar as they have two independent paths to process shape and color, producing two scores for each of these attributes.…”

“…Contrary to the advances on sketch-based querying, methods dealing with sketches, including color, like those shown in Fig. 2, have not followed the same direction, having still few works facing this challenging problem [1], [17]. Therefore, our contribution in this work is to propose a quadruplet-based convolutional network architecture that allows us to incorporate an extra visual attribute to the learned feature space.…”

A Systematic Literature Review of Deep Learning Approaches for Sketch-Based Image Retrieval: Datasets, Metrics, and Future Directions

“…[10], [12], [18], [26], [27], [28], [48], [51], [66], [67], [68], [69], [70], [71], [72], [73] RNN+CNN [30], [31], [32], [74], [75] [41] used sketches and natural images to co-train CNNs, prior to which a specific image scaling method and a multi-angle voting scheme were designed for image data to be used together for SBIR. Bui, et al [18] proposed a triplet ranked CNN for SBIR to learn embeddings between sketches and images with significantly improved performance.…”

“…Considering the limitations of digital manga retrieval, [69] constructed an interactive manga retrieval system based on sketches by extracting features of sketches and manga images using two differently trained CNNs. Xia, et al [28] utilize the same and homogeneous three-branch Triplet network and implement a ranking method that accounts for both shape and color matching to accomplish fine-grained retrieval. The use of edge maps in instance-level SBIR systems poses challenges, requiring pre-training of significant edge map data and sensitivity to edge map quality.…”