SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

Yang, Lumin; Zhuang, Jiajie; Fu, Hongbo; Wei, Xiangzhi; Zhou, Kun; Zheng, Youyi

doi:10.1145/3450284

Cited by 47 publications

(23 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, to utilize the structure information of the sketch, we build the sketch query as a graph. Inspired by the graph neural network utilized in the sketch segmentation task, we adopt a similar structure as [37]. The Graph Convolutional Network (GCN) is composed of several modules of EdgeConv [35] connected in a residual manner, as shown in Fig.…”

Section: A Feature Extractionmentioning

confidence: 99%

I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches

Hua¹,

Cheang²,

Fu³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we are interested in the problem of generating target grasps by understanding freehand sketches. The sketch is useful for the persons who cannot formulate language and the cases where a textual description is not available on the fly. However, very few works are aware of the usability of this novel interactive way between humans and robots. To this end, we propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Due to the inherent ambiguity of sketches with abstract details, we take the advantage of the graph by incorporating the structure of the sketch to enhance the representation ability. This graph-represented sketch is further validated to improve the generalization of the network, capable of learning the sketch-queried grasp detection by using a small collection (around 100 samples) of hand-drawn sketches. Additionally, our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications. Experiments on the multi-object VMRD and GraspNet-1Billion datasets demonstrate the good generalization of the proposed method. The physical robot experiments confirm the utility of our method in object-cluttered scenes 1 .

show abstract

Section: A Feature Extractionmentioning

confidence: 99%

I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches

Hua¹,

Cheang²,

Fu³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In the past few years, exciting developments have been made that explore GCNs capabilities for various vision tasks including image classification [5], captioning [31], image understanding [1], action recognition [13], 3D object detection [34], and shape analysis [26]. Yet, until recently, a few attempts [30,29] started to apply GCNs on sketch embedding. The existing visual sparsity and spatial structure of sketch strokes are naturally compatible with graphical representations.…”

Section: Related Workmentioning

confidence: 99%

SketchLattice: Latticed Representation for Sketch Manipulation

Qi¹,

Su²,

Chowdhury³

et al. 2021

Preprint

View full text Add to dashboard Cite

The key challenge in designing a sketch representation lies with handling the abstract and iconic nature of sketches. Existing work predominantly utilizes either, (i) a pixelative format that treats sketches as natural images employing off-the-shelf CNN-based networks, or (ii) an elaborately designed vector format that leverages the structural information of drawing orders using sequential RNN-based methods. While the pixelative format lacks intuitive exploitation of structural cues, sketches in vector format are absent in most cases limiting their practical usage. Hence, in this paper, we propose a lattice structured sketch representation that not only removes the bottleneck of requiring vector data but also preserves the structural cues that vector data provides. Essentially, sketch lattice is a set of points sampled from the pixelative format of the sketch using a lattice graph. We show that our lattice structure is particularly amenable to structural changes that largely benefits sketch abstraction modeling for generation tasks. Our lattice representation could be effectively encoded using a graph model, that uses significantly fewer model parameters (13.5 times lesser) than existing state-of-the-art. Extensive experiments demonstrate the effectiveness of sketch lattice for sketch manipulation, including sketch healing and image-to-sketch synthesis.

show abstract

“…Possibly this is due to the data representation not being optimized along the network parameters for the task. Yang et al [8] apply graph convolutional networks for semantic segmentation at the stroke level to extensions of the QuickDraw data [19,20]. For an in-depth treatment of drawing recognition, we refer the reader to the recent survey by Xu et.…”

Section: Related Workmentioning

confidence: 99%

“…The answer to this question is highly context sensitive and requires reasoning at the local (i.e., stroke) and global (i.e., the diagram or sketch) level. Existing work has been focused on the recognition [1][2][3] and generation of handwritten text [4,5] or the modelling of entire drawings [6][7][8][9] from the Quick, Draw! dataset [10].…”

Section: Introductionmentioning

confidence: 99%

CoSE: Compositional Stroke Embeddings

Aksan,

Deselaers,

Tagliasacchi

et al. 2020

Preprint

View full text Add to dashboard Cite

We present a generative model for stroke-based drawing tasks which is able to model complex free-form structures. While previous approaches rely on sequencebased models for drawings of basic objects or handwritten text, we propose a model that treats drawings as a collection of strokes that can be composed into complex structures such as diagrams (e.g., flow-charts). At the core of the approach lies a novel auto-encoder that projects variable-length strokes into a latent space of fixed dimension. This representation space allows a relational model, operating in latent space, to better capture the relationship between strokes and to predict subsequent strokes. We demonstrate qualitatively and quantitatively that our proposed approach is able to model the appearance of individual strokes, as well as the compositional structure of larger diagram drawings. Our approach is suitable for interactive use cases such as auto-completing diagrams.

show abstract

SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

Cited by 47 publications

References 27 publications

I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches

I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches

SketchLattice: Latticed Representation for Sketch Manipulation

CoSE: Compositional Stroke Embeddings

Contact Info

Product

Resources

About