CoSE: Compositional Stroke Embeddings

Abstract

We present a generative model for stroke-based drawing tasks which is able to model complex free-form structures. While previous approaches rely on sequencebased models for drawings of basic objects or handwritten text, we propose a model that treats drawings as a collection of strokes that can be composed into complex structures such as diagrams (e.g., flow-charts). At the core of the approach lies a novel auto-encoder that projects variable-length strokes into a latent space of fixed dimension. This representation space allows a relational model, operating in latent space, to better capture the relationship between strokes and to predict subsequent strokes. We demonstrate qualitatively and quantitatively that our proposed approach is able to model the appearance of individual strokes, as well as the compositional structure of larger diagram drawings. Our approach is suitable for interactive use cases such as auto-completing diagrams.

Overcoming Failures of Imagination in AI Infused System Development and Deployment

“…-'harms' to mean impediments to technology deployment -'bad intentions' to be of users and not of technology developers -'harm' only in relation to war, government, or 'mass disaster' -or using terms like 'reliable', 'secure' without specifying for who "our work can bring both beneficial and harmful impacts and it really depends on the motivation of the users" (Hu et al, 2020) and "[the work is] academic in nature, and does not pose foreseeable risks regarding defense, security, and other sensitive fields." (Aksan et al, 2020;Wang et al, 2020b;Jiang et al, 2020) Outsourcing the ethical responsibility to others or other stages of technology deployment by ignoring theoretical or technical affordances for misuse and instead referencing biased inputs, engineering mistakes or malicious uses "there exist risks that some engineers [can] deliberately use the algorithm [in a way that would] harm the performance of the designed system" (Hu et al, 2020) Confusing technical advances with positive impact, by -assuming adoption of technical solutions to constitute a benefit -failing to question assumptions behind performance metrics -treating impact statement as a 'sales pitch' "Further extensions include applying [the method] in robotics. Machine learning for robotics is increasingly growing as a field and has potential of revolutionizing technology in the unprecedented way."…”

Coarse-to-Fine Generative Modeling for Graphic Layouts

“…Vector image generation, e.g., sketches, strokes and icons, catches attention until very recently, despite that raster image generation has achieved great success (Radford, Metz, and Chintala 2015;Zhu et al 2017;Arjovsky, Chintala, and Bottou 2017). For example, SketchRNN (Ha and Eck 2017) models all strokes in a sketch as a sequence; Sketchformer (Ribeiro et al 2020) leverages Transformer to learn longer term temporal structure in the stroke sequence; DeepSVG (Carlier et al 2020) disentangles high-level shapes from the low-level commands to reconstruct complex icons; and CoSE (Aksan et al 2020) factors local appearance of a stroke from the global structure of the drawing to model stroke-based data. As graphic layouts have different data structures with aforementioned vector images, recent progress on them cannot be directly adopted.…”

NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation

“…Arguably, PixelCNN [ Van den Oord et al, 2016] can be viewed as an extreme case of this model class that generates one pixel at a time conditioned on previously generated ones without considering a latent space. There are also stroke based generative models like SPIRAL [Ganin et al, 2018], Cose [Aksan et al, 2020], and SketchEmbedNet [Wang et al, 2020]. SPIRAL generates images through a sequence of strokes while Cose and SketchEmbedNet focus on generating sketch images.…”