Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation

Abstract

Procedural models are being widely used to synthesize scenes for graphics, gaming, and to create (labeled) synthetic datasets for ML. In order to produce realistic and diverse scenes, a number of parameters governing the procedural models have to be carefully tuned by experts. These parameters control both the structure of scenes being generated (e.g. how many cars in the scene), as well as parameters which place objects in valid configurations. Meta-Sim aimed at automatically tuning parameters given a target collection of real images in an unsupervised way. In Meta-Sim2, we aim to learn the scene structure in addition to parameters, which is a challenging problem due to its discrete nature. Meta-Sim2 proceeds by learning to sequentially sample rule expansions from a given probabilistic scene grammar. Due to the discrete nature of the problem, we use Reinforcement Learning to train our model, and design a feature space divergence between our synthesized and target images that is key to successful training. Experiments on a real driving dataset show that, without any supervision, we can successfully learn to generate data that captures discrete structural statistics of objects, such as their frequency, in real images. We also show that this leads to downstream improvement in the performance of an object detector trained on our generated dataset as opposed to other baseline simulation methods. Project page: https://nv-tlabs.github.io/meta-sim-structure/.

ProbRobScene: A Probabilistic Specification Language for 3D Robotic Manipulation Environments

“…Another common approach to generating environments is using a probabilistic grammar. This technique has been used to create generative models of buildings [18], traffic scenes [19], and even plants [20]. However, while these methods excel at generating inputs with rich structure, phrasing a generator as a grammar makes it difficult to specify the inherently relational constraints between objects that our domain requires.…”

DriveGAN: Towards a Controllable High-Quality Neural Simulation

“…The goal of data-driven simulation is to learn simulators given observations from the environment to be simulated. Meta-Sim [27,9] learns to produce scene parameters in a synthetic scene. LiDARSim [42] leverages deep learning and physics engine to produce LiDAR point clouds.…”

“…Li-darSim [42] used a catalog of annotated 3D scenes to sample layouts into which reconstructed objects obtained from a large number of recorded drives are placed, in the quest to achieve diversity for training and testing a LIDAR-based perception system. [27,9,51], on the other hand, learn to synthesize road-scene 3D layouts directly from images without supervision. These works do not model the dynamics of the environment and object behaviors.…”