Language-driven synthesis of 3D scenes from scene databases

Abstract

We introduce a novel framework for using natural language to generate and edit 3D indoor scenes, harnessing scene semantics and text-scene grounding knowledge learned from large annotated 3D scene databases. The advantage of natural language editing interfaces is strongest when performing semantic operations at the sub-scene level, acting on groups of objects. We learn how to manipulate these sub-scenes by analyzing existing 3D scenes. We perform edits by first parsing a natural language command from the user and transforming it into a semantic scene graph that is used to retrieve corresponding sub-scenes from the databases that match the command. We then augment this retrieved sub-scene by incorporating other objects that may be implied by the scene context. Finally, a new 3D scene is synthesized by aligning the augmented sub-scene with the user's current scene, where new objects are spliced into the environment, possibly triggering appropriate adjustments to the existing scene arrangement. A suggestive modeling interface with multiple interpretations of user commands is used to alleviate ambiguities in natural language. We conduct studies comparing our approach against both prior text-to-scene work and artist-made scenes and find that our method significantly outperforms prior work and is comparable to handmade scenes even when complex and varied natural sentences are used.

PlanIT

“…What graph representation maps most naturally to this language? There exists prior work in language-based scene creation [Chang et al 2015[Chang et al , 2014, including recent work that uses a graph-based intermediate representation [Ma et al 2018a]. However, it constructs scenes by retrieving parts of scenes from a database; new possibilities are opened up by a system that can synthesize truly new scenes from a partial graph.…”

“…Followup work has used undirected factor graphs learned from annotated RGB-D images [Kermani et al 2016], relation graphs between objects learned from human activity annotations [Fu et al 2017], and directed graphical models with Gaussian mixtures for modeling arrangement patterns [Paul Henderson 2018]. Other work has focused on conditioning the scene generation using input from RGB-D frames [Chen et al 2014], 2D sketches of the scene [Xu et al 2013], natural language text [Chang et al 2015;Ma et al 2018b], or activity predictions on RGB-D reconstructions [Fisher et al 2015].…”

Active Arrangement of Small Objects in 3D Indoor Scenes

“…Users can define their preference for small object arrangement interactively with our framework by manipulating the spatial relations between any two small objects. However, at the current stage, our framework cannot support advanced spatial relations such as "surrounded by" [37], which cannot be simply interpreted as multiple pairwise relations. It would be interesting to include these relations by proposing another effective data form, e.g.…”

Learning Generative Models of 3D Structures

“…In terms of input, earlier works on probabilistic models, e.g., [FRS∗12], generates a new scene by taking a random sample from a learned distribution, while recent works on deep generative neural networks, e.g., [LPX∗19], can produce a novel scene from a random noise vector. The input can also be a hand sketch [XCF∗13], a photograph [ISS17, LZW∗15], natural language commands [MGPF∗18], or human actions/activities [FLS∗15, MLZ∗16]. In terms of output, while most methods have been designed to generate room layouts with 3D furniture objects, some methods learn to produce floor or building plans [MSK10, WFT∗19].…”