LiDAR-as-Camera for End-to-End Driving

Abstract

The core task of any autonomous driving system is to transform sensory inputs into driving commands. In end-to-end driving, this is achieved via a neural network, with one or multiple cameras as the most commonly used input and low-level driving commands, e.g., steering angle, as output. However, simulation studies have shown that depth-sensing can make the end-to-end driving task easier. On a real car, combining depth and visual information can be challenging due to the difficulty of obtaining good spatial and temporal alignment of the sensors. To alleviate alignment problems, Ouster LiDARs can output surround-view LiDAR images with depth, intensity, and ambient radiation channels. These measurements originate from the same sensor, rendering them perfectly aligned in time and space. The main goal of our study is to investigate how useful such images are as inputs to a self-driving neural network. We demonstrate that such LiDAR images are sufficient for the real-car road-following task. Models using these images as input perform at least as well as camera-based models in the tested conditions. Moreover, LiDAR images are less sensitive to weather conditions and lead to better generalization. In a secondary research direction, we reveal that the temporal smoothness of off-policy prediction sequences correlates with the actual on-policy driving ability equally well as the commonly used mean absolute error.

General-Purpose Deep Learning Detection and Segmentation Models for Images from a Lidar-Based Camera Sensor

“…While we did not study invariability of object detection or segmentation of the same class of objects across different environments (e.g., indoors and outdoors) or environmental conditions (e.g., light rain or fog, day or night), we could assume, based on the works in the literature [ 16 ], that the data characteristics did not change significantly. Indeed, one of the key benefits of lidar-generated images is that they are not affected by environmental conditions.…”

“…A relevant recent work in the literature is [ 16 ], where the authors presented a novel dataset of lidar-generated images with the same lidar-as-a-camera sensor that we used in this paper. The work in [ 16 ] showed the potential of these images, as they remained almost invariant across seasonal changes and environmental conditions. For example, unpaved roads could be perceived in very similar ways in summer weather, snow cover, or light rain.…”

“…A sample of the data used in this work is shown in Figure 1 . We refer the reader to existing dataset papers with this type of data for a more in-depth characterization of the different types of images that the Ouster lidar sensors can generate [ 16 ].…”

The Effects of Speed and Delays on Test-Time Performance of End-to-End Self-Driving

“…In our previous work [ 29 ], we noticed that the driving speed differed between the data collection drives (including data for off-policy evaluation) and deployment (on-policy testing). In particular, we deployed steering-only models on a real vehicle at speeds 0.5 to 0.8 times the speed the human used in the given GPS location.…”

Robust Multi-Modal Multi-LiDAR-Inertial Odometry and Mapping for Indoor Environments

“…Our proposed methods show robust performance in long corridor environments and survive in narrow spaces where 180°U-turn occurs. LiDAR technology has advanced rapidly in recent years, with new sensors able of generating image-like data in addition to point clouds [7], [8], and new solid-state LiDAR sensors that offer dense 3D point clouds with non-repetitive scan patterns, while at the same time lowering the cost [9], [10]. Despite the clear advantages of solid-state LiDARs, the naturally narrow horizontal FoV leads, in a similar way to monocular pinhole camera systems, to the sensing volume being blocked by objects, or even entirely occupied by a near wall, resulting in an insufficient number of feature points to estimate a 6degree-of-freedom (6-DoF) pose.…”