3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

Abstract

A comprehensive semantic understanding of a scene is important for many applications -but in what space should diverse semantic information (e.g., objects, scene categories, material types, texture, etc.) be grounded and what should be its structure? Aspiring to have one unified structure that hosts diverse types of semantics, we follow the Scene Graph paradigm in 3D, generating a 3D Scene Graph. Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e.g., class, material, and other attributes), rooms (e.g., scene category, volume, etc.) and cameras (e.g., location, etc.), as well as the relationships among these entities.However, this process is prohibitively labor heavy if done manually. To alleviate this we devise a semi-automatic framework that employs existing detection methods and enhances them using two main constraints: I. framing of query images sampled on panoramas to maximize the performance of 2D detectors, and II. multi-view consistency enforcement across 2D detections that originate in different camera locations.

TSM: Topological Scene Map for Representation in Indoor Environment Understanding

“…Expressing the environment with 3D information preserved supports the scene graph to record the environment detail, but it requires powerful computation capabilities. In [13], a framework for constructing a 3D scene semantic map is proposed. The map constructed by this framework is composed of four layers, which is more in line with human thinking and perception.…”

“…Among the scene graph, each node represents the object and attributes, and each edge represents the relation between the objects. Based on [12], [13], MIT SPARK laboratory combines with the previous semantic mapping work, visual-inertial odometry, deep learning, and other methods to construct a scene graph of a dynamic 3D environment [11]. They propose a more comprehensive 3D semantic SGG framework, which adds the detection and tracking modules for dynamic targets, thus some of the impacts of dynamic changes is eliminated.…”

“…Since the scene graph constructed from a single image is not specific enough, the scene graph generated from multiple images may miss some objects or repeatedly detect some objects. We refer to the method mentioned in [11]- [13] to build the scene graph from RGB-D videos.…”

“…The 3D semantic map construction framework proposed in [11] is a masterpiece of semantic map research, which expresses the environment with five levels: Metric-Semantic Mesh, Objects and Agents, Places and Structures, Rooms, Building. Besides, the framework applies the tracking and detection module for dynamic targets, which eliminate the effects of dynamic changes [12], [13]. However, this framework has strict requirements on hardware and the construction process of the map is complicated.…”

Variable photorealistic image synthesis for training dataset generation

“…There are also works that use variety of approaches to scene description and generation, such as domain specific languages [23], scene graphs [24], stochastic grammars [25] for scenes description and generation.…”