Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs

Rosinol, Antoni; Violette, Andrew; Abate, Marcus; Hughes, Nathan; Chang, Yun Sil; Shi, Jingnan; Gupta, Arjun K.; Carlone, Luca

doi:10.48550/arxiv.2101.06894

Cited by 5 publications

(54 citation statements)

References 157 publications

(262 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Armeni et al [4] model the environment as a graph including low-level geometry (i.e., a metric-semantic mesh), objects, rooms, and camera locations. Rosinol et al [49,50] augment the model with a topological map of places, as well as a layer describing moving entities in the environment. The approaches in [4,49,50] are designed for offline operation.…”

Section: Related Workmentioning

confidence: 99%

“…Rosinol et al [49,50] augment the model with a topological map of places, as well as a layer describing moving entities in the environment. The approaches in [4,49,50] are designed for offline operation. Other papers focus on reconstructing a graph of objects and their relations [26,63,67].…”

Section: Related Workmentioning

confidence: 99%

“…After extracting the 3D mesh within the active window, we segment objects by performing Euclidean clustering of the 3D metric-semantic mesh vertices; in particular, we independently cluster vertices belonging to each semantic class. As in [50], the result of the Euclidean clustering is then used to estimate a centroid and bounding box for each putative object. These are then associated with previous object nodes inside the active window, and any putative objects that do not correspond to a previous object node are added as new nodes.…”

Section: Real-time Incremental 3d Scene Graph Layers Constructionmentioning

confidence: 99%

“…Datasets. We utilize two datasets for our experiments: uHu-mans2 (uH2) [50] and SidPac. The uH2 dataset is a Unitybased simulated dataset [50] that includes three scenes: a small apartment, an office, and a subway station.…”

Section: A Experimental Setupmentioning

confidence: 99%

“…Unfortunately, ESDFs scale poorly in the size of the environment [43]. Moreover, the extraction of places and rooms in [49,50] involves batch algorithms that process the entire ESDF, whose computational cost grows over time and is incompatible with real-time operation. Finally, the ESDF is reconstructed from the robot trajectory estimate; however, during operation the trajectory estimate keeps changing in response to loop closures; therefore, the approaches in [49,50] would need to rebuild the scene graph from scratch after every loop closure, which again clashes with real-time operation.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hughes¹,

Chang²,

Carlone³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction (from low-level geometry to high-level semantics including objects, places, rooms, buildings, etc.) and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory.This paper describes the first real-time Spatial Perception engINe (SPIN), a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance, to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed SPIN into a highly parallelized architecture, named Hydra, that combines fast early and midlevel perception processes (e.g., local mapping) with slower highlevel perception (e.g., global optimization of the scene graph). We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods, while running online.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Real-time Incremental 3d Scene Graph Layers Constructionmentioning

confidence: 99%

Section: A Experimental Setupmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hughes¹,

Chang²,

Carlone³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hughes¹,

Chang²,

Carlone³

2022

Robotics: Science and Systems XVIII

Self Cite

View full text Add to dashboard Cite

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction (from low-level geometry to high-level semantics including objects, places, rooms, buildings, etc.) and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory.This paper describes a real-time Spatial Perception System, a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop realtime algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed Spatial Perception System into a highly parallelized architecture, named Hydra 1 , that combines fast early and mid-level perception processes (e.g., local mapping) with slower high-level perception (e.g., global optimization of the scene graph). We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods despite running online.

show abstract

Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks

Ravichandran¹,

Peng²,

Hughes³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Representations are crucial for a robot to learn effective navigation policies. Recent work has shown that mid-level perceptual abstractions, such as depth estimates or 2D semantic segmentation, lead to more effective policies when provided as observations in place of raw sensor data (e.g., RGB images). However, such policies must still learn latent three-dimensional scene properties from mid-level abstractions. In contrast, high-level, hierarchical representations such as 3D scene graphs explicitly provide a scene's geometry, topology, and semantics, making them compelling representations for navigation. In this work, we present a reinforcement learning framework that leverages high-level hierarchical representations to learn navigation policies. Towards this goal, we propose a graph neural network architecture and show how to embed a 3D scene graph into an agent-centric feature space, which enables the robot to learn policies for low-level action in an end-to-end manner. For each node in the scene graph, our method uses features that capture occupancy and semantic content, while explicitly retaining memory of the robot trajectory. We demonstrate the effectiveness of our method against commonly used visuomotor policies in a challenging object search task. These experiments and supporting ablation studies show that our method leads to more effective object search behaviors, exhibits improved long-term memory, and successfully leverages hierarchical information to guide its navigation objectives.

show abstract

Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs

Cited by 5 publications

References 157 publications

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks

Contact Info

Product

Resources

About