Deep Manifold Alignment for Mid-Grain Sketch Based Image Retrieval

Bui, Tien D.; Ribeiro, Leonardo Sampaio Ferraz; Ponti, Moacir Antonelli; Collomosse, John

doi:10.1007/978-3-030-20893-6_20

Cited by 10 publications

(6 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Category-level sketch-image retrieval has been extensively studied by existing works [4], [14]- [16]. There are three mainstream methodologies in the field of category-level SBIR:…”

Section: Related Work a Category Level Sbirmentioning

confidence: 99%

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Wang¹,

Sapkota²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Fine-Grained Sketch-Based Image Retrieval (FG-SBIR) aims at finding a specific image from a large gallery given a query sketch. Despite the widespread applicability of FG-SBIR in many critical domains (e.g., crime activity tracking), existing approaches still suffer from a low accuracy while being sensitive to external noises such as unnecessary strokes in the sketch. The retrieval performance will further deteriorate under a more practical on-the-fly setting, where only a partially complete sketch with only a few (noisy) strokes are available to retrieve corresponding images. We propose a novel framework that leverages a uniquely designed deep reinforcement learning model that performs a dual-level exploration to deal with partial sketch training and attention region selection. By enforcing the model's attention on the important regions of the original sketches, it remains robust to unnecessary stroke noises and improve the retrieval accuracy by a large margin. To sufficiently explore partial sketches and locate the important regions to attend, the model performs bootstrapped policy gradient for global exploration while adjusting a standard deviation term that governs a locator network for local exploration. The training process is guided by a hybrid loss that integrates a reinforcement loss and a supervised loss. A dynamic ranking reward is developed to fit the on-the-fly image retrieval process using partial sketches. The extensive experimentation performed on three public datasets shows that our proposed approach achieves the state-of-the-art performance on partial sketch based image retrieval.

show abstract

“…Category-level sketch-image retrieval has been extensively studied by existing works [4], [14]- [16]. There are three mainstream methodologies in the field of category-level SBIR:…”

Section: Related Work a Category Level Sbirmentioning

confidence: 99%

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Wang¹,

Sapkota²,

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Category-level SBIR: Category-level sketch-photo retrieval is now well studied [3,43,2,6,5,47,39,9,24,4,23]. Contemporary research directions can be broadly classified into traditional SBIR, zero-shot SBIR and sketchimage hashing.…”

Section: Related Workmentioning

confidence: 99%

“…Contemporary research directions can be broadly classified into traditional SBIR, zero-shot SBIR and sketchimage hashing. In traditional SBIR [3,5,4,2], object classes are common to both training and testing. Whereas zero-shot SBIR [47,6,9,24] asks models to generalise across disjoint training and testing classes in order to alleviate annotation costs.…”

Section: Related Workmentioning

confidence: 99%

Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval

Bhunia

Yang

Hospedales

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early-retrieval efficiency over state-of-theart methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.

show abstract

“…Figure 6 provides a taxonomy for SBIR. From the perspective of evaluation criterion, SBIR can be divided into conventional/coarse-grained SBIR (i.e., category-level SBIR), mid-grained [133], and fine-grained SBIR (i.e., instance-level SBIR). FG-SBIR is essentially a kind of instance-level retrieval [134].…”

Section: Sketch-photo Retrievalmentioning

confidence: 99%

Deep Learning for Free-Hand Sketch: A Survey

Xu¹

2020

Preprint

View full text Add to dashboard Cite

Free-hand sketches are highly hieroglyphic and illustrative, which have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly more popular. The prosperity of deep learning has also immensely promoted the research for the free-hand sketch. This paper presents a comprehensive survey of the free-hand sketch oriented deep learning techniques. The main contents of this survey include: (i) The intrinsic traits and domain-unique challenges of the free-hand sketch are discussed, to clarify the essential differences between free-hand sketch and other data modalities, e.g., natural photo. (ii) The development of the free-hand sketch community in the deep learning era is reviewed, by surveying the existing datasets, research topics, and the state-of-the-art methods via a detailed taxonomy. (iii) Moreover, the bottlenecks, open problems, and potential research directions of this community have also been discussed to promote the future works.

show abstract

Deep Manifold Alignment for Mid-Grain Sketch Based Image Retrieval

Cited by 10 publications

References 25 publications

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval

Deep Learning for Free-Hand Sketch: A Survey

Contact Info

Product

Resources

About