Neural Scene Representation for Locomotion on Structured Terrain

Abstract

We propose a learning-based method to reconstruct the local terrain for locomotion with a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the algorithm estimates the topography in the robot's vicinity. The raw measurements from these cameras are noisy and only provide partial and occluded observations that in many cases do not show the terrain the robot stands on. Therefore, we propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement. The model consists of a 4D fully convolutional network on point clouds that learns the geometric priors to complete the scene from the context and an auto-regressive feedback to leverage spatio-temporal consistency and use evidence from the past. The network can be solely trained with synthetic data, and due to extensive augmentation, it is robust in the real world, as shown in the validation on a quadrupedal robot, ANYmal, traversing challenging settings. We run the pipeline on the robot's onboard low-power computer using an efficient sparse tensor implementation and show that the proposed method outperforms classical map representations.

ANYmal parkour: Learning agile navigation for quadrupedal robots

“…It ingests noisy and heavily occluded point clouds of the scene coming from depth cameras and LiDAR to produce a 3D estimate of the terrain centered around the robot as well as a compact latent representation of the scene. Similar to ( 37 ), the point cloud data are spatially and temporally processed using a fully convolutional encoder-decoder network architecture. The encoder takes in the point cloud and compresses it into a compact representation that is used by the decoder to complete the missing information and filter out noise.…”

“…In each occupied voxel, a feature describes the position of the centroid of the points that fall within that voxel, and the features of unoccupied voxels were set to 0. Despite the sparse implementation used in ( 37 ), the library did not scale well to the typical batch sizes required for reinforcement learning. Unexpectedly, a dense formulation can handle such a large batch size with satisfactory speeds, but this comes at the cost of high memory requirements (approximately 45 GB of GPU memory for a batch size of 4096).…”

“…The reconstructed point cloud can then be recovered by pruning the cells whose occupancy probability is below a user-defined threshold. Contrary to ( 37 ), no skip connections were used to produce a more informative latent that the navigation module can directly use. Although this might limit the generalization performance, we found that it worked well for our task with randomized parkour worlds.…”

“…Similar to (31), the navigation module then learned to steer and switch between those skills. For the perception module, we took inspiration from (37) to reconstruct the environment in 3D from point cloud data. We augmented the method with a multi-resolution scheme to combine a higher-resolution map near the robot with a coarser larger-scale map further away.…”

City-scale Incremental Neural Mapping with Three-layer Sampling and Panoptic Representation

“…Contrary to explicit representations, implicitly defined, continuous, differentiable shape representations parameterized by MLP have emerged as a powerful paradigm. It is memory-efficient and easily deals with a wide variety of surface topologies without resolution limitation, enabling downstream tasks ranging from robotic perception [9] and 3D reconstruction to navigation [1]. Recently, research about RGBD-based incremental implicit mappings has made significant progress [18], [32], but they are all used for indoor scene reconstruction.…”