Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Abstract

Neural Cellular Automata (NCAs) have been proven effective in simulating morphogenetic processes, the continuous construction of complex structures from very few starting cells. Recent developments in NCAs lie in the 2D domain, namely reconstructing target images from a single pixel or infinitely growing 2D textures. In this work, we propose an extension of NCAs to 3D, utilizing 3D convolutions in the proposed neural network architecture. Minecraft is selected as the environment for our automaton since it allows the generation of both static structures and moving machines. We show that despite their simplicity, NCAs are capable of growing complex entities such as castles, apartment blocks, and trees, some of which are composed of over 3,000 blocks. Additionally, when trained for regeneration, the system is able to regrow parts of simple functional machines, significantly expanding the capabilities of simulated morphogenetic systems.

Collective Intelligence for Deep Learning: A Survey of Recent Developments

“…Over time, a consensus will be formed as to which digit is the most likely pixel, but interestingly, disagreements may result depending on the location of the pixel, especially if the image is intentionally drawn to represent different digits. 64 's formulation enabled the regeneration of not only Minecraft buildings, trees, but also simple functional machines in the game such as worm-like creatures that can even regenerate into two distinct creatures when cut in half.…”

“…This approach is also applicable outside of pure generative domains, and can also be applied to the construction of artificial agents in active environments such as Minecraft. Sudhakaran et al 64 trained neural CAs to grow complex entities from Minecraft such as castles, apartment blocks, and trees, some of which are composed of thousands of blocks. Aside from regeneration, their system is able to regrow parts of simple functional machines (such as a virtual creature in the game), and they demonstrate a morphogenetic creature grow into two distinct creatures when cut in half in the virtual world (See Figure 3).…”

“…Neural CAs have also been applied to the regeneration of Minecraft entities. Sudhakaran et al64 's formulation enabled the regeneration of not only Minecraft buildings, trees, but also simple functional machines in the game such as worm-like creatures that can even regenerate into two distinct creatures when cut in half.…”

The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning

“…Using modern deep learning tools, recent work demonstrates that neural CA, or self-organized neural networks performing only local computation, can generate (and re-generate) coherent images [54] and voxel scenes [69,83], and even perform image classification [60]. Self-organizing neural network agents have been proposed in the RL domain [10,11,58,59], with recent work demonstrating that shared local policies at the actuator level [42], through communicating with their immediate neighbors, can learn a global coherent policy for continuous control locomotion tasks.…”

Variational Neural Cellular Automata

“…Neural Cellular Automata (NCA) are CA where the cell states are vectors, and the rules of the CA are parameterized and learned by neural networks (Mordvintsev et al, 2020;Nichele et al, 2017;Stanley & Miikkulainen, 2003;Wulff & Hertz, 1992). NCAs have been shown to learn to generate images, 3D structures, and even functional artifacts that are capable of regenerating when damaged (Mordvintsev et al, 2020;Sudhakaran et al, 2021). While these results are impressive, the NCAs can only generate (and regenerate) the single artifact it is trained on, lacking the diverse generative properties of current probabilistic generative models.…”