Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions

Wald, Johanna; Dhamo, Helisa; Navab, Nassir; Tombari, Federico

doi:10.1109/cvpr42600.2020.00402

Cited by 118 publications

(147 citation statements)

References 37 publications

Supporting

Mentioning

141

Contrasting

Order By: Relevance

“…Armeni et al (2016), Mura et al (2014), and Ochmann et al (2014) are closest to our proposal, but in contrast to Armeni et al (2016) we do not rely on a Manhattan World assumption, and in contrast to Mura et al (2014) and Ochmann et al (2014) we operate on a mesh model. Recently, Wald et al (2020) proposed to learn from point clouds a 3D semantic scene graph that focuses on representing semantically meaningful inter-instance relationships.…”

Section: Related Workmentioning

confidence: 99%

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Rosinol

Violette

Abate

et al. 2021

The International Journal of Robotics Research

101

View full text Add to dashboard Cite

Humans are able to form a complex mental model of the environment they move in. This mental model captures geometric and semantic aspects of the scene, describes the environment at multiple levels of abstractions (e.g., objects, rooms, buildings), includes static and dynamic entities and their relations (e.g., a person is in a room at a given time). In contrast, current robots’ internal representations still provide a partial and fragmented understanding of the environment, either in the form of a sparse or dense set of geometric primitives (e.g., points, lines, planes, and voxels), or as a collection of objects. This article attempts to reduce the gap between robot and human perception by introducing a novel representation, a 3D dynamic scene graph (DSG), that seamlessly captures metric and semantic aspects of a dynamic environment. A DSG is a layered graph where nodes represent spatial concepts at different levels of abstraction, and edges represent spatiotemporal relations among nodes. Our second contribution is Kimera, the first fully automatic method to build a DSG from visual–inertial data. Kimera includes accurate algorithms for visual–inertial simultaneous localization and mapping (SLAM), metric–semantic 3D reconstruction, object localization, human pose and shape estimation, and scene parsing. Our third contribution is a comprehensive evaluation of Kimera in real-life datasets and photo-realistic simulations, including a newly released dataset, uHumans2, which simulates a collection of crowded indoor and outdoor scenes. Our evaluation shows that Kimera achieves competitive performance in visual–inertial SLAM, estimates an accurate 3D metric–semantic mesh model in real-time, and builds a DSG of a complex indoor environment with tens of objects and humans in minutes. Our final contribution is to showcase how to use a DSG for real-time hierarchical semantic path-planning. The core modules in Kimera have been released open source.

show abstract

Section: Related Workmentioning

confidence: 99%

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Rosinol

Violette

Abate

et al. 2021

The International Journal of Robotics Research

101

View full text Add to dashboard Cite

show abstract

“…Thereby, the deep learning technology has recently been adopted by some researchers. Wald et al [40] proposed two PointNet architectures for the extraction of objects and their spatial relationships and exploited a Graph Convolutional Networks to process the acquired object-object relationships. Although the deep learning techniques have shown prospects in the extraction of certain spatial relations, obtaining effective features of the complex spatial relationships is still difficult.…”

Section: The Acquisition Of Spatial Relationshipsmentioning

confidence: 99%

“…However, it is difficult to build up a fixed parameter model for training due to the complexity of 3D spatial relationships. To fix the problem, Wald et al [40] recently tried to use deep learning techniques to train and predict spatial relations. The deep learning-based method showed prospects in the extraction of certain spatial relationships.…”

Section: Introductionmentioning

confidence: 99%

Slice-Guided Components Detection and Spatial Semantics Acquisition of Indoor Point Clouds

Wang

2022

Sensors

View full text Add to dashboard Cite

Extracting indoor scene components (i.e., the meaningful parts of indoor objects) and obtaining their spatial relationships (e.g., adjacent, in the left of, etc.) is crucial for scene reconstruction and understanding. At present, the detection of indoor scene components with complex shapes is still challenging. To fix the problem, a simple yet powerful slice-guided algorithm is proposed. The key insight is that slices of indoor scene components always have similar profiles no matter if the components are simple-shaped or complex-shaped. Specifically, we sliced the indoor scene model into many layers and transformed each slice into a set of two-dimensional (2D) profiles by resampling. After that, we clustered 2D profiles from neighbor slices into different components on the base of spatial proximity and similarity. To acquire the spatial relationships between indoor scene components, an ontology was constructed to model the commonsense knowledge about the semantics of indoor scene components and their spatial relationships. Then the spatial semantics of the relationships between indoor scene components were inferred and a semantic graph of spatial relationship (SGSR) was yielded to represent them. The experimental results demonstrate that our method can effectively detect complex-shaped indoor scene components. The spatial relationships between indoor components can be exactly acquired as well.

show abstract

“…Scene graphs have also been used in 3D scene understanding [21]. Kim et al [22] merged multiple local scene graphs, from single-camera images with overlapping views, and combines them into a global scene graph based on the similar features between the nodes of the local scene graphs.…”

Section: Related Workmentioning

confidence: 99%

Multi-Camera Video Scene Graphs for Surveillance Videos Indexing and Retrieval

Patel¹,

Yao²,

Yu³

et al. 2021

2021 IEEE International Conference on Image Processing (ICIP)

View full text Add to dashboard Cite

Modern video surveillance systems often consist of multiple cameras capturing continuous videos. However, existing systems do not fully take advantage of shared scene semantics across multiple video streams, and hence querying the captured videos to find an event or object of interest can be a time-consuming task. In this paper, we propose a compact representation of objects and their relationships in the scene under surveillance, over time, and across multiple cameras, using a combined global spatio-temporal scene graph. The same objects that appear in multiple cameras are stored only once, reducing the storage required and speeding up the query when compared to storing the scene graph from each camera individually. Our experiments show that in a 5-camera system our proposed representation can speed up querying time by a factor of 3.9 times.

show abstract

Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions

Cited by 118 publications

References 37 publications

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Kimera: From SLAM to spatial perception with 3D dynamic scene graphs

Slice-Guided Components Detection and Spatial Semantics Acquisition of Indoor Point Clouds

Multi-Camera Video Scene Graphs for Surveillance Videos Indexing and Retrieval

Contact Info

Product

Resources

About