Global-to-local generative model for 3D shapes

Abstract

We introduce a generative model for 3D man-made shapes. The presented method takes a global-to-local (G2L) approach. An adversarial network (GAN) is built first to construct the overall structure of the shape, segmented and labeled into parts. A novel conditional auto-encoder (AE) is then augmented to act as a part-level refiner. The GAN, associated with additional local discriminators and quality losses, synthesizes a voxel-based model, and assigns the voxels with part labels that are represented in separate channels. The AE is trained to amend the initial synthesis of the parts, yielding more plausible part geometries. We also introduce new means to measure and evaluate the performance of an adversarial generative model. We demonstrate that our global-to-local generative model produces significantly better results than a plain three-dimensional GAN, in terms of both their shape variety and the distribution with respect to the training data.

Composite Shape Modeling via Latent Space Factorization

“…A simple approach for such factorization is to split the dimensions of the n-dimensional embedding space into K coordinate groups, each group representing a certain semantic part-embedding. In this case, the full shape embedding is a concatenation of part embeddings, an approach explored in [39]. This, however, puts a hard constraint on the dimensionality of part embeddings, and thus also on the representation capacity of each part embedding subspace.…”

“…two must be implicitly or explicitly modeled and learned by the system. Examples of such structure-aware shape representation-learning are [24,20,39,43].…”

“…However, the existing approaches for shape modeling, while being part aware at the intermediate stages of the system, still ultimately operate on the low-dimensional representations of the whole shape. For example, [24,39] use a Variational Autoencoder (VAE) [16] to learn generative part-aware models of man-made shapes, but the latent spaces of the VAEs correspond to complete shapes, with entangled latent factors corresponding to different semantic parts. Therefore, these and other existing approaches cannot perform part-level shape manipulation, such as single part replacement, part interpolation, or part-level shape synthesis.…”

Multi-theme generative adversarial terrain amplification

“…Generative models have also been extensively used in the past few years in the area of computer graphics, e.g., 3D shape modeling and generation [Liu et al 2017;Wang et al 2018], rendering hair and 3D scenes [Feygina et al 2018;, uid ow generation [Xie et al 2018], food image generation [Fujieda et al 2017], generating mass models [Kelly et al 2018], animated facial image generation [Geng et al 2018], transferring shape deformation [Gao et al 2018], photo-to-caricature translation [Cao et al 2018], face swapping [Natsume et al 2018], and sele-to-avatar translation [Nagano et al 2018].…”

TileGAN

“…While traditional GANs generate images starting from a random vector, the GAN training can be extended to the problem of image-to-image translation using either paired or unpaired training data Isola et al 2017;Zhu et al 2017a,b]. In computer graphics, recent papers apply GANs to the synthesis of caricatures of human faces [Cao et al 2018], the synthesis of human avatars from a single image [Nagano et al 2018], texture and geometry synthesis of building details [Kelly et al 2018], surface-based modeling of shapes [Ben-Hamu et al 2018] and the volumetric modeling of shapes [Wang et al 2018a]. The most related problem to our work is the problem of terrain synthesis [Guérin et al 2017].…”