Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network

Bui, Tien D.; Ribeiro, Leonardo Sampaio Ferraz; Ponti, Moacir Antonelli; Collomosse, John

doi:10.1016/j.cviu.2017.06.007

Cited by 70 publications

(55 citation statements)

References 43 publications

(67 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The addition of fc layers to create cross-modal embedding (-R) slightly improves (importantly, does not degrade) the intermediate raster embedding R available via [6]. The method significantly outperforms recent triplet SBIR approaches [28,5]. Note that the S-I and S-S figures are non-comparable; they search different datasets.…”

Section: Evaluating Cross-modal Searchmentioning

confidence: 98%

“…7 (top). We perform two ablations to our proposed LiveSketch (LS) system: 1) querying with rasterized versions of the QD-345 queries (-R) using the proposed embedding S; 2) querying with rasterized queries in the intermediate embedding R (-R-I) which degenerates to [6]; we also baseline against two further recent SBIR techniques: the unshared triplet GoogleNet-V1 architecture proposed by Sangkloy et al [28], and the triplet edgemap approach of Bui et al [5]. We compute class-and instance-level precision for all queries resulting in 345 × 15 × 5 =∼ 26K MTurk annotations.…”

Section: Evaluating Cross-modal Searchmentioning

confidence: 99%

“…We perform 3 ablations to our method (LS): 1) non-interactive (-NI), users are not offered sketch suggestions; 2) sketches are rasterized (-R) rather than processed as vector queries; 3) as -R but searching within intermediate embedding R which degenerates to [6] (-R-I). We also baseline against [28,5]. Fig.…”

Section: Evaluating Iterative Retrievalmentioning

confidence: 99%

See 2 more Smart Citations

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

Collomosse

Bui

Jin

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

LiveSketch is a novel algorithm for searching large image collections using hand-sketched queries. LiveSketch tackles the inherent ambiguity of sketch search by creating visual suggestions that augment the query as it is drawn, making query specification an iterative rather than one-shot process that helps disambiguate users' search intent. Our technical contributions are: a triplet convnet architecture that incorporates an RNN based variational autoencoder to search for images using vector (stroke-based) queries; real-time clustering to identify likely search intents (and so, targets within the search embedding); and the use of backpropagation from those targets to perturb the input stroke sequence, so suggesting alterations to the query in order to guide the search. We show improvements in accuracy and time-to-task over contemporary baselines using a 67M image corpus.

show abstract

Section: Evaluating Cross-modal Searchmentioning

confidence: 98%

Section: Evaluating Cross-modal Searchmentioning

confidence: 99%

See 1 more Smart Citation

LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search

Collomosse

Bui

Jin

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

“…An early example was SketchANet [45], which performed sketch recognition using Alexnet [24]. More recently, triplet convolutional neural networks have gained interest as they have the capacity to deal with deep embedding spaces [9]. Improving the image similarity metric is a main challenge as these triplet architectures are used to measure similarities between images and sketches [10].…”

Section: D Sketching For Retrievalmentioning

confidence: 99%

Mixing realities for sketch retrieval in Virtual Reality

Giunchi

James

Degraen

et al. 2019

Proceedings of the 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry

View full text Add to dashboard Cite

Drawing tools for Virtual Reality (VR) enable users to model 3D designs from within the virtual environment itself. These tools employ sketching and sculpting techniques known from desktopbased interfaces and apply them to hand-based controller interaction. While these techniques allow for mid-air sketching of basic shapes, it remains difficult for users to create detailed and comprehensive 3D models. In our work, we focus on supporting the user in designing the virtual environment around them by enhancing sketch-based interfaces with a supporting system for interactive model retrieval. Through sketching, an immersed user can query a database containing detailed 3D models and replace them into the virtual environment. To understand supportive sketching within a virtual environment, we compare different methods of sketch interaction, i.e., 3D mid-air sketching, 2D sketching on a virtual tablet, 2D sketching on a fixed virtual whiteboard, and 2D sketching on a real tablet. Our results show that 3D mid-air sketching is considered to be a more intuitive method to search a collection of models while the addition of physical devices creates confusion due to the complications of their inclusion within a virtual environment. While we pose our work as a retrieval problem for 3D models of chairs, our results can be extrapolated to other sketching tasks for virtual environments.

show abstract

“…This set of values can be seen as a feature vector with 2048 dimensions to be an input for another classifier, such as the SVM for example. If a more compact representation is needed, one can use dimensionality reduction methods or quantization based on PCA [51] or Product Quantization [52], [53].…”

Section: Beyond Classification: Fine-tuning Feature Extraction Anmentioning

confidence: 99%