Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

Bhunia, Ayan Kumar; Koley, Subhadeep; Khilji, Abdullah Faiz Ur Rahman; Sain, Aneeshan; Chowdhury, Pinaki Nath; Xiang, Tao; Song, Yi-Zhe

doi:10.1109/cvpr52688.2022.00107

Cited by 38 publications

(8 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sketches are ideal interfaces between human and computer software: we can easily modify sketches by drawing several lines, letting computers handle more tedious artistic work (such as image retrieval [71] or full-color image generation [72, 73]) based on the sketches we provide (see [74] for review). Sketch generation, the AI problem we study in this work, may further simplify the artistic interaction between humans and computers.…”

Section: Discussionmentioning

confidence: 99%

Top-down generation of low-resolution representations improves visual perception and imagination

2021

Preprint

View full text Add to dashboard Cite

According to analysis-by-synthesis theories of perception, the primary visual cortex (V1) reconstructs visual stimuli through top-down pathway, and higher-order cortex reconstructs V1 activity. Experiments also found that neural representations are generated in a top-down cascade during visual imagination. What code does V1 provide higher-order cortex to reconstruct or simulate to improve perception or imaginative creativity? What unsupervised learning principles shape V1 for reconstructing stimuli so that V1 activity eigenspectrum is power-law with close-to-1 exponent? Using computational models, we reveal that reconstructing the activities of V1 complex cells facilitate higher-order cortex to form representations smooth to shape morphing of stimuli, improving perception and creativity. Power-law eigenspectrum with close-to-1 exponent results from the constraints of sparseness and temporal slowness when V1 is reconstructing stimuli, at a sparseness strength that best whitens V1 code and makes the exponent most insensitive to slowness strength. Our results provide fresh insights into V1 computation.

show abstract

Section: Discussionmentioning

confidence: 99%

Top-down generation of low-resolution representations improves visual perception and imagination

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, our model trained on ShoeV2 generalises well to all unseen sketch styles, yielding compelling results. Robustness and Sensitivity: The free-flow style of amateur sketching is likely to introduce irrelevant noisy strokes [10]. To prove our model's robustness to noise, during testing, we gradually add synthetic noisy strokes [57] onto clean input sketches.…”

Section: Performance Analysis and Discussionmentioning

confidence: 99%

“…We also briefly showcase two potential downstream tasks our generation model enables: fine-grained sketch-based image retrieval (FG-SBIR), and precise semantic editing. On the former, we show how FG-SBIR, a well-studied task in the sketch community [10,[71][72][73], can be reduced to an image (generated) to image retrieval task, and that a simple nearest-neighbour model based on VGG-16 [78] features can already surpass state-of-the-art. On the latter, we demonstrate how precise local editing can be done that is more fine-grained than those possible with text and attributes.…”

Section: Introductionmentioning

confidence: 99%

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

Koley¹,

Bhunia²,

Sain³

et al. 2023

Preprint

View full text Add to dashboard Cite

Edgemap Sketch Sketch Sketch Sketch Sketch Existing Methods Proposed Method Figure 1. (a) Set of photos generated by the proposed method. (b) While existing methods can generate faithful photos from perfectly pixel-aligned edgemaps, they fall short drastically in case of highly deformed and sparse free-hand sketches. In contrast, our autoregressive sketch-to-photo generation model produces highly photorealistic outputs from highly abstract sketches.

show abstract

“…Introduced as a deep triplet-ranking based siamese network [70] for learning a joint sketchphoto manifold, FG-SBIR was improvised via attentionbased modules with a higher order retrieval loss [60], textual tags [12,59], hybrid cross-domain generation [43], hierarchical co-attention [51] and reinforcement learning [9]. Furthermore, sketch-traits like style-diversity [52], datascarcity [5] and redundancy of sketch-strokes [6] were addressed in favor of retrieval. Towards generalising to novel classes, while [42] modelled a universal manifold of prototypical visual sketch traits embedding sketch and photo, [8] adapted to new classes via some supporting sketch-photo pairs.…”

Section: Related Workmentioning

confidence: 99%

“…While off-the-shelf CLIP itself has zero-shot image retrieval potential [30,61] where someone can feed the category level query as 'a photo of a [query]', it raises a question -how much is a category-level querysketch beneficial over text-keyword based query? Attending to sketch's specialty in modelling fine-grained [6,51,70] details hence, we go beyond category-level ZS-SBIR [16,18] to a more practical and long-standing research problem of cross-category fine-grained ZS-SBIR [44].…”

Section: Prompt Learning For Zs-sbirmentioning

confidence: 99%

CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not

Sain¹,

Bhunia²,

Chowdhury³

et al. 2023

Preprint

View full text Add to dashboard Cite

In this paper, we leverage CLIP for zero-shot sketch based image retrieval (ZS-SBIR). We are largely inspired by recent advances on foundation models and the unparalleled generalisation ability they seem to offer, but for the first time tailor it to benefit the sketch community. We put forward novel designs on how best to achieve this synergy, for both the category setting and the fine-grained setting ("all"). At the very core of our solution is a prompt learning setup. First we show just via factoring in sketch-specific prompts, we already have a category-level ZS-SBIR system that overshoots all prior arts, by a large margin (24.8%) -a great testimony on studying the CLIP and ZS-SBIR synergy. Moving onto the fine-grained setup is however trickier, and requires a deeper dive into this synergy. For that, we come up with two specific designs to tackle the fine-grained matching nature of the problem: (i) an additional regularisation loss to ensure the relative separation between sketches and photos is uniform across categories, which is not the case for the gold standard standalone triplet loss, and (ii) a clever patch shuffling technique to help establishing instance-level structural correspondences between sketchphoto pairs. With these designs, we again observe significant performance gains in the region of 26.9% over previous state-of-the-art. The take-home message, if any, is the proposed CLIP and prompt learning paradigm carries great promise in tackling other sketch-related tasks (not limited to ZS-SBIR) where data scarcity remains a great challenge. Project page: https:// aneeshan95.github.io/ Sketch LVM/

show abstract

Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

Cited by 38 publications

References 39 publications

Top-down generation of low-resolution representations improves visual perception and imagination

Top-down generation of low-resolution representations improves visual perception and imagination

Picture that Sketch: Photorealistic Image Generation from Abstract Sketches

CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not

Contact Info

Product

Resources

About