Intra-category sketch-based image retrieval by matching deformable part models

Abstract

Figure 1: Comparison of traditional text-based image retrieval, conventional SBIR, and the proposed fine-grained SBIR framework.Introduction Sketches are known to be able to capture object appearance and structure more intuitively and precisely than bare texts. However, to date the main focus of sketch-based image retrieval (SBIR) has been on retrieving photos of the same category, overlooking an important property of sketches -they can capture fine-grained variations of objects such as pose (standing vs. sitting) and iconic pattern (textures on a cow's body). By further leveraging this descriptive power of sketches, in this paper, for the first time we introduce fine-grained SBIR. That is to study how sketches can be used to differentiate fine-grained variations of objects for retrieval, specifically pose variations. Figure 1 contrasts text-based image retrieval and conventional SBIR with our proposed fine-grained SBIR. Methodology Key to this problem is introducing a mid-level sketch representation that not only captures object pose, but also possesses the ability to traverse sketch and photo domains. Specifically, we learn deformable part-based model (DPM) [3] to discover and encode the various poses and parts in sketch and image domains independently, and employ graph matching [1] to establishing the correspondence between DPMs from different domains. The DPM is a two-layer structure, composed of root filter and part filters. We denote DPM as M = (r, G), where r = (w, h, f ) specifies the width w, height h and global appearance feature of the root filter; and G = (V, E, A) represents the star graph composed of the part filters. For the star graph G, V represents a set of nodes, E, edges, and A, attributes. Our matching objective for DPM accounts for both appearance and geometric information encoded in DPM, as well as both layers of representation, i.e., root filter r and part filter star graph G. Given two DPMs M R and M T , the similarity function is defined as:

Sketch Me That Shoe

“…We also compare with the pose-centric fine-grained retrieval model [16], testing our model on their dataset. Specifically, we fine-tune our pre-trained chair model on their dataset, and test using their evaluation setting.…”

“…A bag-of-words (BOW) representation combined with some form of edge detection from photo images are often employed to bridge the domain gap. The only previous work that attempted to address the fine-grained SBIR problem is that of [16], which is based on deformable part-based model (DPM) and graph matching. However, their definition of fine-grain is very different from ours -a sketch is considered to be a match to a photo if the objects depicted look similar, i.e.…”

“…Therefore, with few exceptions [22,11], most existing SBIR datasets were created by combining overlapping categories of sketches and photos, which means only categorylevel SBIR is possible. The 'semi'-fine-grained dataset in [16] was created by selecting similar-looking sketch-photo pairs from the TU-Berlin and and Pascal VOC datasets. For each of 14 categories, there are 6 sketches and 60 images -much smaller than ours, and too small to apply state of the art deep learning techniques.…”

Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval

“…The problem of fine-grained SBIR was first proposed in [15], which employed a deformable part-based model (DPM) representation and graph matching. More recently, the FG-SBIR problem is tackled by deep learning [46,31] which aims to learn both feature representation and cross-domain matching function jointly.…”

“…Among them, using a sketch to retrieve a specific object instance, or fine-grained sketch-based image retrieval (FG-SBIR) [15,46,31] is of particular interest due to its potential in commercial applications such as searching online product catalogues for shoes, furniture, and hand- * These authors contributed equally to this work traced outline (human edgemap) edgemap human free--hand sketch Figure 1. FG-SBIR is challenging due to the misalignment of the domains (left) and subtle local appearance differences between a true match photo and a visually similar incorrect match (right).…”