Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval

Abstract

Fine-grained sketch-based image retrieval (FG-SBIR) addresses the problem of retrieving a particular photo instance given a user's query sketch. Its widespread applicability is however hindered by the fact that drawing a sketch takes time, and most people struggle to draw a complete and faithful sketch. In this paper, we reformulate the conventional FG-SBIR framework to tackle these challenges, with the ultimate goal of retrieving the target photo with the least number of strokes possible. We further propose an on-the-fly design that starts retrieving as soon as the user starts drawing. To accomplish this, we devise a reinforcement learning based cross-modal retrieval framework that directly optimizes rank of the ground-truth photo over a complete sketch drawing episode. Additionally, we introduce a novel reward scheme that circumvents the problems related to irrelevant sketch strokes, and thus provides us with a more consistent rank list during the retrieval. We achieve superior early-retrieval efficiency over state-of-theart methods and alternative baselines on two publicly available fine-grained sketch retrieval datasets.

Deep Reinforced Attention Regression for Partial Sketch Based Image Retrieval

“…While there has been much research effort in leveraging reinforcement learning (RL) [19] in various computer vision problems, RL is not commonly used for SBIR with few exceptions. Ayan et al propose a partial sketch training procedure, where RL is leveraged to trade-off between sketch recognisability and the number of strokes to observe [13]. It introduces a discount weight and negative rewards for the number of strokes.…”

“…• Triplet + Vanilla RL: To support partial sketch matching, this model combines the triplet network for image embedding with a simple policy network [28] to generate the final sketch representation , which can then be matched with the image embedding for retrieval. • Triplet + PPO: In this model [13], the simple policy network is replaced by an improved version of the PPO network [29].…”

“…Therefore, our first metric evaluates the percentage of complete sketches with their true matching images appearing in the top-q list, which is referred to as acc@q. To assess the retrieval performance on partial sketches, the metric should encourage early retrieval [13]. Thus, we compute the mean area under the plot of 1/rank versus the completion degree of a sketch, referred to as area under inverse rank or short for AUIR.…”

“…Existing work to provide image retrieval based on partial sketches is fairly scarce. One exception is the model that combines a triplet network with a reinforcement learning (RL) agent to predict the embedding of a partial sketch on the fly [13]. In this model, a partial sketch is regarded as a state of a particular time step from a full episode.…”

“…The proposed framework aims to deal with partial sketches while being robust to noisy stokes to further advance the state of the art in FG-SBIR and support the practical on-the-fly image retrieval. While a RL agent adopted by existing works can capture the temporal dynamics among different stokes in a sketch to predict the embedding of a final sketch [13], the incomplete sketches containing noisy strokes along with a potentially large image repository make the reward signal very sparse and delayed. The proposed attention mechanism aims to identify the most important regions in the sketch while ignoring the unnecessary noisy strokes.…”