ShapeAssembly

Abstract

Manually authoring 3D shapes is difficult and time consuming; generative models of 3D shapes offer compelling alternatives. Procedural representations are one such possibility: they offer high-quality and editable results but are difficult to author and often produce outputs with limited diversity. On the other extreme are deep generative models: given enough data, they can learn to generate any class of shape but their outputs have artifacts and the representation is not editable. In this paper, we take a step towards achieving the best of both worlds for novel 3D shape synthesis. First, we propose ShapeAssembly, a domain-specific "assembly-language" for 3D shape structures. ShapeAssembly programs construct shape structures by declaring cuboid part proxies and attaching them to one another, in a hierarchical and symmetrical fashion. ShapeAssembly functions are parameterized with continuous free variables, so that one program structure is able to capture a family of related shapes. We show how to extract ShapeAssembly programs from existing shape structures in the PartNet dataset. Then, we train a deep generative model, a hierarchical sequence VAE, that learns to write novel ShapeAssembly programs. Our approach leverages the strengths of each representation: the program captures the subset of shape variability that is interpretable and editable, and the deep generative model captures variability and correlations across shape collections that is hard to express procedurally. We evaluate our approach by comparing the shapes output by our generated programs to those from other recent shape structure synthesis models. We find that our generated shapes are more plausible and physically-valid than those of other methods. Additionally, we assess the latent spaces of these models, and find that ours is better structured and produces smoother interpolations. As an application, we use our generative model and differentiable program interpreter to infer and fit shape programs to unstructured geometry, such as point clouds.

Differentiable 3D CAD Programs for Bidirectional Editing

“…There has also been increased effort to reconstruct CAD programs from existing geometry [DIP*18; NWP*18; WPL*20], infer programs to construct assemblies [JBX*20], infer higher‐level operations from those programs [JCG*21], and re‐write CAD programs to expose meaningful parameters [NWA*20]. While none of these methods propose solutions to interactive manipulation, they are critical for enabling wide application of our work: can be modified to generate and refine a CAD program suitable for input to our system, growing the potential application of bidirectional editing beyond human‐written programs.…”

Automatic Differentiable Procedural Modeling

“…L‐system grammar from a 2D image of branching structures has been inferred by Guo et al [GJB ∗ 20], and both Hu et al [HDR19] and Guo et al [GHYZ20] use learning methods to infer parameters of procedural materials. Jones et al [JBX ∗ 20] learn to generate programs written in a domain‐specific language that generate 3D shapes. Liu et al [LVW ∗ 15] generate variations of an existing model by splitting it, and reassembling its parts into a plausible model.…”

DAG amendment for inverse control of parametric shapes

“…[Ganin et al 2018] considers that the input to a renderer is a program rather than a model and trains its machine learning model so. [Jones et al 2020] learns shape generation programs (i.e. parametric shapes).…”