Multi-Graph Transformer for Free-Hand Sketch Recognition

Xu, Peng; Joshi, Chaitanya K.; Bresson, Xavier

doi:10.48550/arxiv.1912.11258

Cited by 20 publications

(20 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The core concept of the multi-head self-attention layer is the selfattention mechanism [35,37], which is beneficial to capture the dependencies between span representation pairs regardless of their distance in the sequences. Because self-attention is mainly based on linear operations, the function of the position-wise feed-forward network is to do non-linear transformation and generate high-level abstract representations.…”

Section: Bidirectional Transformermentioning

confidence: 99%

InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping

Liu,

Qin,

et al. 2021

Preprint

View full text Add to dashboard Cite

E-commerce companies have to face abnormal sellers who sell potentially-risky products. Typically, the risk can be identified by jointly considering product content (e.g., title and image) and seller behavior. This work focuses on the behavior feature extraction as behavior sequences can provide valuable clues for the risk discovery by reflecting the sellers' operation habits. Traditional feature extraction techniques heavily depend on domain experts and adapt poorly to new tasks. In this paper, we propose a self-supervised method InfoBehavior to automatically extract meaningful representations from ultra-long raw behavior sequences instead of the costly feature selection procedure. InfoBehavior utilizes Bidirectional Transformer as feature encoder due to its excellent capability in modeling long-term dependency. However, it is intractable for commodity GPUs because the time and memory required by Transformer grow quadratically with the increase of sequence length. Thus, we propose a hierarchical grouping strategy to aggregate ultra-long raw behavior sequences to length-processable high-level embedding sequences. Moreover, we introduce two types of pretext tasks. Sequence-related pretext task defines a contrastive-based training objective to correctly select the masked-out coarse-grained/finegrained behavior sequences against other "distractor" behavior sequences; Domain-related pretext task designs a classification training objective to correctly predict the domain-specific statistical results of anomalous behavior. We show that behavior representations from the pre-trained InfoBehavior can be directly used or integrated with features from other side information to support a wide range of downstream tasks. Experimental results demonstrate that InfoBehavior significantly improves the performance of Product Risk Management and Intellectual Property Protection.

show abstract

Section: Bidirectional Transformermentioning

confidence: 99%

InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping

Liu,

Qin,

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Sketch Representation: Besides the traditionally used convolutional neural networks [39], there have been some recent advancements towards robust representation learning for sketches, e.g., SketchRNN [9], transformer-based architecture [36]. Although we have chosen to utilize CNN (ResNet)-based feature encoders in this work, our proposed localization framework can seamlessly integrate more advanced sketch representation learning methods in the feature extraction modules.…”

Section: Attention In Deep Neural Networkmentioning

confidence: 99%

Sketch-Guided Object Localization in Natural Images

Tripathi¹,

Dani²,

Mishra³

et al. 2020

Preprint

View full text Add to dashboard Cite

http://visual-computing.in/sketch-guided-object-localization/ Abstract. We introduce the novel problem of localizing all the instances of an object (seen or unseen during training) in a natural image via sketch query. We refer to this problem as sketch-guided object localization. This problem is distinctively different from the traditional sketch-based image retrieval task where the gallery set often contains images with only one object. The sketch-guided object localization proves to be more challenging when we consider the following: (i) the sketches used as queries are abstract representations with little information on the shape and salient attributes of the object, (ii) the sketches have significant variability as they are hand-drawn by a diverse set of untrained human subjects, and (iii) there exists a domain gap between sketch queries and target natural images as these are sampled from very different data distributions. To address the problem of sketch-guided object localization, we propose a novel cross-modal attention scheme that guides the region proposal network (RPN) to generate object proposals relevant to the sketch query. These object proposals are later scored against the query to obtain final localization. Our method is effective with as little as a single sketch query. Moreover, it also generalizes well to object categories not seen during training and is effective in localizing multiple object instances present in the image. Furthermore, we extend our framework to a multi-query setting using novel feature fusion and attention fusion strategies introduced in this paper. The localization performance is evaluated on publicly available object detection benchmarks, viz. MS-COCO and PASCAL-VOC, with sketch queries obtained from 'Quick, Draw!'. The proposed method significantly outperforms related baselines on both single-query and multi-query localization tasks.

show abstract

“…With the prevalence of touchscreen devices in recent years, more and more sketches are spreading on the internet, bringing new challenges to the sketch research community. This has led to a flourishing in sketch-related research [1], including sketch recognition [2], [3], sketch-based image retrieval (SBIR) [4], [5], sketch segmentation [6], sketch generation [7], [8], etc. However, sketches are essentially different from natural photos.…”

Section: Introductionmentioning

confidence: 99%

On Learning Semantic Representations for Million-Scale Free-Hand Sketches

Xu,

Huang,

Yuan

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

In this paper, we study learning semantic representations for million-scale free-hand sketches. This is highly challenging due to the domain-unique traits of sketches, e.g., diverse, sparse, abstract, noisy. We propose a dual-branch CNN-RNN network architecture to represent sketches, which simultaneously encodes both the static and temporal patterns of sketch strokes. Based on this architecture, we further explore learning the sketch-oriented semantic representations in two challenging yet practical settings, i.e., hashing retrieval and zero-shot recognition on million-scale sketches. Specifically, we use our dualbranch architecture as a universal representation framework to design two sketch-specific deep models: (i) We propose a deep hashing model for sketch retrieval, where a novel hashing loss is specifically designed to accommodate both the abstract and messy traits of sketches. (ii) We propose a deep embedding model for sketch zero-shot recognition, via collecting a largescale edge-map dataset and proposing to extract a set of semantic vectors from edge-maps as the semantic knowledge for sketch zero-shot domain alignment. Both deep models are evaluated by comprehensive experiments on million-scale sketches and outperform the state-of-the-art competitors.

show abstract

Multi-Graph Transformer for Free-Hand Sketch Recognition

Cited by 20 publications

References 40 publications

InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping

InfoBehavior: Self-supervised Representation Learning for Ultra-long Behavior Sequence via Hierarchical Grouping

Sketch-Guided Object Localization in Natural Images

On Learning Semantic Representations for Million-Scale Free-Hand Sketches

Contact Info

Product

Resources

About