Perceiving like a Bat: Hierarchical 3D Geometric–Semantic Scene Understanding Inspired by a Biomimetic Mechanism

Abstract

Geometric–semantic scene understanding is a spatial intelligence capability that is essential for robots to perceive and navigate the world. However, understanding a natural scene remains challenging for robots because of restricted sensors and time-varying situations. In contrast, humans and animals are able to form a complex neuromorphic concept of the scene they move in. This neuromorphic concept captures geometric and semantic aspects of the scenario and reconstructs the scene at multiple levels of abstraction. This article seeks to reduce the gap between robot and animal perception by proposing an ingenious scene-understanding approach that seamlessly captures geometric and semantic aspects in an unexplored environment. We proposed two types of biologically inspired environment perception methods, i.e., a set of elaborate biomimetic sensors and a brain-inspired parsing algorithm related to scene understanding, that enable robots to perceive their surroundings like bats. Our evaluations show that the proposed scene-understanding system achieves competitive performance in image semantic segmentation and volumetric–semantic scene reconstruction. Moreover, to verify the practicability of our proposed scene-understanding method, we also conducted real-world geometric–semantic scene reconstruction in an indoor environment with our self-developed drone.

SonoNERFs: Neural Radiance Fields applied to Biological Echolocation Systems allow 3D Scene Reconstruction Through Perceptual Prediction

“…While it has been observed that bats can predict aspects of the scene by accumulating sensor data [60], to the best of our knowledge, no concrete model on how this prediction might operate has been proposed in previous works. Other technical systems have been proposed to produce 3D scene reconstruction and semantic interpretation, [61], [62], but these proposed techniques utilize a teaching modality like LIDARs or cameras to perform a form of modality translation. Our SonoNERF model relies solely on acoustic data without the need for an additional supervision modality.…”

“…Our SonoNERF model relies solely on acoustic data without the need for an additional supervision modality. Furthermore, reference [62] does not use an acoustic sensing modality, causing the title to be somewhat misleading. Our approach follows the approach called ‘self-supervised learning’ which has received much attention in the recent years [63], [64].…”

SonoNERFs: Neural Radiance Fields Applied to Biological Echolocation Systems Allow 3D Scene Reconstruction through Perceptual Prediction

“…What, in the opinion of the authors, is not so trivial is the fact that 3D scene reconstruction emerges when solving a completely different task, namely predicting sensor data for a novel scene, and whereas it has been observed that bats can predict aspects of the scene by accumulating sensor data [ 58 ], to the best of our knowledge, no concrete model on how this prediction might operate has been proposed in previous works. Other technical systems have been proposed to produce 3D scene reconstruction and semantic interpretation [ 59 , 60 ], but these proposed techniques utilize a teaching modality like LIDARs or cameras to perform a form of modality translation. Our SonoNERF model relies solely on acoustic data without the need for an additional supervision modality.…”

“…Our SonoNERF model relies solely on acoustic data without the need for an additional supervision modality. Furthermore, reference [ 60 ] does not use an acoustic sensing modality, causing the title to be somewhat misleading. Our approach follows the approach called ‘self-supervised learning’, which has received much attention in recent years [ 61 , 62 ].…”