Sketchformer: Transformer-Based Representation for Sketched Structure

Abstract

Sketchformer is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer effectively addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks, when compared against baseline representations driven by LSTM sequence to sequence architectures: SketchRNN and derivatives. We show that sketch reconstruction and interpolation are improved significantly by the Sketchformer embedding for complex sketches with longer stroke sequences.

Light-SRNet: a lightweight dual-attention feature fusion network for hand-drawn sketch recognition

“…This method achieves good recognition performance but endures a complex training process and a huge number of model parameters. Ribeiro et al 11 constructed a sketch recognition network based on the transformer structure that encodes sketches into feature vectors and uses the stroke sequence of sketches as input to the network, enhancing the network's ability to learn the stroke sequence of complex sketches. Jain et al 12 further designed the TransSketchNet in accordance with the transformer structure, which improves the network's ability to extract more valuable features by entirely using the stroke sequences and the attention mechanism.…”

“…This method achieves good recognition performance but endures a complex training process and a huge number of model parameters. Ribeiro et al 11 . constructed a sketch recognition network based on the transformer structure that encodes sketches into feature vectors and uses the stroke sequence of sketches as input to the network, enhancing the network’s ability to learn the stroke sequence of complex sketches.…”

“…On the basis of existing approaches, 6 , 7 , 19 , 23 – 47 we introduce the dual-attention mechanism followed by the CNN backbone, so that the proposed model can fully use the sketch features in the learning processes. Rather than using the transformer structure 10 – 12 or embedding the attention mechanism into certain modules, we adopt the attention mechanism itself directly to keep the lightweight nature of the model.…”

“…However, the aforementioned methods rarely fully utilize the extracted sketch features. To address this issue, researchers have incorporated attention mechanisms 9 into networks, such as multi-graph transformer (MGT), 10 SketchFormer, 11 and TransSketchNet 12 . Although these methods focus on the features through attention mechanisms, they all require a complex computational process.…”

Text-Conditioned Transformer for Automatic Pronunciation Error Detection

“…The ASR performance is reported to be better by combining the CTC loss with the attention mechanism [28] or using the Transformer structure [14,15]. In particular, the Transformer structure, which is originally designed to handle the natural language processing (NLP) problems [29,30], has been successfully utilized in several other domains, such as computer vision (CV) [31,32], and speech-related tasks including text to speech (TTS) [33,34,18,19], voice conversion (VC) [35], and ASR [12,13].…”