A Zero-Shot Framework for Sketch Based Image Retrieval

Yelamarthi, Sasi Kiran; Reddy, M Shiva Krishna; Mishra, Ashish; Mittal, Anurag

doi:10.1007/978-3-030-01225-0_19

Cited by 147 publications

(122 citation statements)

References 33 publications

Supporting

Mentioning

121

Contrasting

Order By: Relevance

“…Comparison: Table 2 provides comparisons of our full model results against those of the state-of-the-art. We report a comparative study with regard to two methods presented in Section 2, namely ZSIH [28] and CVAE [36]. Note that we have not been able to reproduce the ZSIH model due to lack of technical implementation details and the code being unavailable.…”

Section: Model Discussionmentioning

confidence: 99%

“…On one hand, Shen et al [28] proposed to set aside 25 random classes as a test set whereas the training is performed in the rest 100 classes. On the other hand, Yelamarthi et al [36] proposed a different partition of 104 train classes and 21 test classes in order to make sure that test is not present in the 1,000 classes of ImageNet. Its main limitation for the task of ZS-SBIR is its fine-grained nature, i.e., each sketch has a corresponding photo that was used as reference at drawing time.…”

Section: Quickdraw-extended Datasetmentioning

confidence: 99%

“…Manual filtering is performed to remove outliers. Moreover, following the idea introduced in [36] for the Sketchy-Extended dateset, we provide a test split which forces that test classes are not present in Im-ageNet in case of using pre-trained models. Finally, this dataset consists of 330,000 sketches and 204,000 photos moving towards a large-scale retrieval.…”

Section: The Datasetmentioning

confidence: 99%

“…Table 2: Comparison against the state-of-the-art with that of the proposed model. Note: the same train and test split are used for all experiments on CVAE [36] and ours. ZSIH [28] did not report the specific details on their split (other than 25 classes were used for testing), and we could not produce their results on QuickDraw-Extended due to the lack of publicly available code.…”

Section: Experimental Validationmentioning

confidence: 99%

“…Note that this evaluation does not consider visually similar drawings that can be considered correct by human users. For the existing datasets, we used the proposed splits in [36,28].…”

Section: Zero-shot Experimental Settingmentioning

confidence: 99%

See 4 more Smart Citations

Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval

Dey¹,

Riba²,

Dutta³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

151

174

View full text Add to dashboard Cite

In this paper, we investigate the problem of zeroshot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330, 000 sketches and 204, 000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research † .

show abstract

Section: Model Discussionmentioning

confidence: 99%

Section: Quickdraw-extended Datasetmentioning

confidence: 99%

Section: The Datasetmentioning

confidence: 99%

Section: Experimental Validationmentioning

confidence: 99%

“…Note that this evaluation does not consider visually similar drawings that can be considered correct by human users. For the existing datasets, we used the proposed splits in [36,28].…”

Section: Zero-shot Experimental Settingmentioning

confidence: 99%

See 3 more Smart Citations

Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval

Dey¹,

Riba²,

Dutta³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

151

174

View full text Add to dashboard Cite

show abstract

Fine-Grained Color Sketch-Based Image Retrieval

Xia

Wang

et al. 2019

Advances in Computer Graphics

View full text Add to dashboard Cite

We propose a novel fine-grained color sketch-based image retrieval (CSBIR) approach. The CSBIR problem is investigated for the first time using deep learning networks, in which deep features are used to represent color sketches and images. A novel ranking method considering both shape matching and color matching is also proposed. In addition, we build a CSBIR dataset with color sketches and images to train and test our method. The results show that our method has better retrieval performance.

show abstract