Building LEGO Using Deep Generative Models of Graphs

Abstract

Generative models are now used to create a variety of high-quality digital artifacts. Yet their use in designing physical objects has received far less attention. In this paper, we advocate for the construction toy, LEGO, as a platform for developing generative models of sequential assembly. We develop a generative model based on graph-structured neural networks that can learn from human-built structures and produce visually compelling designs. Our code is released at: https://github. com/uoguelph-mlrg/GenerativeLEGO.Generative models can now create high-quality and diverse digital artifacts, particularly in the domain of images [2] and text [3]. Generative models are also being applied to physical design, such as computational chemistry and materials but they are less prevalent in architecture, landscape design, or manufacturing. Motivated by the opportunity of generative design of our physical environment, in this paper we explore the use of generative models to modular physical design.Advances in generative models in areas such as images and text is in part due to the availability of data and ease of experimentation. Other areas of machine learning, such as reinforcement learning, have benefited from the availability of simulators. Simulators are conducive to experimentation supporting algorithm development and, at the same time, entertaining and familiar (c.f. the Arcade Learning Environment [4]). In the context of physical design, specifically the sequential assembly of modular structures, we believe that there is an analogue: the plastic construction toy, LEGO. Structures built from LEGO have a number of interesting properties: (i) they are complex enough to approach real-world design; ii) they are familiar and fun enough to attract interest from the generative models community; and iii) there is a huge amount of official and user-created structures available as a source of training data.Learning to sequentially assemble LEGO structures based on human-generated examples is a sequential decision making problem. It can be approached through a number of formalisms familiar to the ML community, including reinforcement learning [5], imitation learning [6], evolutionary design [7,8], and Bayesian optimization [9]. We are unaware of the use of deep generative models for LEGO. However, generative graph models (GGMs), an emerging sub-field of graph representation learning [10], are particularly well suited to this domain.Workshop on machine learning for engineering modeling, simulation and design @ NeurIPS 2020

Break and Make: Interactive Structural Understanding Using LEGO Bricks

“…In contrast to these approaches, recent works have suggested data-driven deep learning approaches for LEGO problems based on generative models of graphs [49], and image to voxel reconstruction [28]. Similar to Break and Make, Chung et al [3] propose a method for assembling LEGO structures from a reference image using interactive learning.…”

Reinforcement Learning for Sequential Assembly of SL-Blocks - Self-interlocking combinatorial design based on Machine Learning

“…Since the problem of sequential assembly belongs to the class of combinatorial optimization problems, research on methods for solving such problems, in general, is highly relevant. In recent years, there has been an increasing interest in applying deep learning techniques to combinatorial optimization problems (Bello et al 2016, Khalil et al 2017 as well as in using intelligent algorithms for construction and assembly tasks (Bapst et al 2019, Thompson et al 2020, Cho et al 2020). However, most of the previous works on intelligent construction focused on automating the entire process end to end, i.e., without any means for the user to influence the design process (Bapst et al 2019, Thompson et al 2020.…”

“…In recent years, there has been an increasing interest in applying deep learning techniques to combinatorial optimization problems (Bello et al 2016, Khalil et al 2017 as well as in using intelligent algorithms for construction and assembly tasks (Bapst et al 2019, Thompson et al 2020, Cho et al 2020). However, most of the previous works on intelligent construction focused on automating the entire process end to end, i.e., without any means for the user to influence the design process (Bapst et al 2019, Thompson et al 2020. While this might result in an over-all optimal design, it also clearly limits the interactivity of the approaches.…”

Parameter Prediction for Unseen Deep Architectures

“…One way to measure the distance between two feature distributions, such as our graph embeddings, is the Fréchet distance (FD) [108]. We compute the FD between graph embeddings of 5000 training architectures and five test subsets (in the similar style as in [109][110][111]): ID-TEST and four OOD subsets (Table 11).…”