DDD20 End-to-End Event Camera Driving Dataset: Fusing Frames and Events with Deep Learning for Improved Steering Prediction

Abstract

Neuromorphic event cameras are useful for dynamic vision problems under difficult lighting conditions. To enable studies of using event cameras in automobile driving applications, this paper reports a new end-to-end driving dataset called DDD20. The dataset was captured with a DAVIS camera that concurrently streams both dynamic vision sensor (DVS) brightness change events and active pixel sensor (APS) intensity frames. DDD20 is the longest event camera end-to-end driving dataset to date with 51h of DAVIS event+frame camera and vehicle human control data collected from 4000km of highway and urban driving under a variety of lighting conditions. Using DDD20, we report the first study of fusing brightness change events and intensity frame data using a deep learning approach to predict the instantaneous human steering wheel angle. Over all day and night conditions, the explained variance for human steering prediction from a Resnet-32 is significantly better from the fused DVS+APS frames (0.88) than using either DVS (0.67) or APS (0.77) data alone.

DSEC: A Stereo Event Camera Dataset for Driving Scenarios

“…ADD [32] 0.1 --2D Bounding Box 1MP Detection [33] 4 -yes 0.9 --2D Bounding Box N-CARS [34] 0.01 --Binary Class DDD17 [35] 0.1 -no 0.1 --Vehicle Control, GPS DDD20 [36] 0.1 -no 0.1 --Vehicle Control, GPS DET [37] 1.0 --Lane Extraction Brisbane-Event-VPR [38] 2 or not applicable respectively. ADD, N-CARS and DET do not feature a frame camera.…”

“…Note that a baseline of above 50 centimeters is consistent with the KITTI datasets [3], [5] and DrivingStereo [6] VPR [38] presents a driving dataset with a color DAVIS346 camera for place recognition. DDD17 [35] and DDD20 [36] are datasets comprising many hours of driving data from a monochrome DAVIS346 camera featuring various vehicle control data. These event camera datasets do not contain stereo event cameras and can thus not be used for the development of stereo matching algorithms.…”

Combining Events and Frames Using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

“…This is why for more complex tasks such as image reconstruction or monocular depth, state-of-the-art methods use a data-driven approach [7], [6], [19], [20], [21].Of these, many rely on recurrent architectures which can leverage long time windows of events for improved prediction [21], [7]. Although there exist many purely event-based learning methods, few address the fusion of images and events [9], [10], [22]. These approaches fuse both modalities by synchronizing and concatenating both inputs and passing them to a standard feed-forward network [9], [10], [22].…”

Neuromorphic Engineering Needs Closed-Loop Benchmarks

“…Alternatively, event-based sensors have been used directly to recreate existing open-loop datasets for handwritten digit recognition (Diehl and Cook, 2015 ; Orchard et al, 2015a ; Cohen et al, 2018 ), object classification (Orchard et al, 2015a ; Serrano-Gotarredona and Linares-Barranco, 2015 ; Cohen et al, 2018 ), autonomous driving (Binas et al, 2017 ; Hu et al, 2020 ), pedestrian detection (Miao et al, 2019 ), pose estimation (Mueggler et al, 2017 ; Calabrese et al, 2019 ), spoken digit classification (Anumula et al, 2018 ), or speaker identification (Ceolini et al, 2020 ) (please refer to Figure 3 or Tables 1 , 2 for a more complete listing of existing datasets and benchmarks).…”