DSEC: A Stereo Event Camera Dataset for Driving Scenarios

Abstract

Once an academic venture, autonomous driving has received unparalleled corporate funding in the last decade. Still, operating conditions of current autonomous cars are mostly restricted to ideal scenarios. This means that driving in challenging illumination conditions such as night, sunrise, and sunset remains an open problem. In these cases, standard cameras are being pushed to their limits in terms of low light and high dynamic range performance. To address these challenges, we propose, DSEC, a new dataset that contains such demanding illumination conditions and provides a rich set of sensory data. DSEC offers data from a wide-baseline stereo setup of two color frame cameras and two highresolution monochrome event cameras. In addition, we collect lidar data and RTK GPS measurements, both hardware synchronized with all camera data. One of the distinctive features of this dataset is the inclusion of high-resolution event cameras. Event cameras have received increasing attention for their high temporal resolution and high dynamic range performance. However, due to their novelty, event camera datasets in driving scenarios are rare. This work presents the first high resolution, large scale stereo dataset with event cameras. The dataset contains 53 sequences collected by driving in a variety of illumination conditions and provides ground truth disparity for the development and evaluation of event-based stereo algorithms.

How to Calibrate Your Event Camera

“…The closest work to ours [12] applies image reconstruction for camera calibration. However, a detailed evaluation of their method and a comparison against other methods was never conducted, thus leaving the question open whether this method is accurate.…”

E$^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action Recognition

“…Thanks to their focus on capturing only variations in the scene, event-based cameras are particularly efficient in ego-centric scenarios, as they drastically reduce the amount of data to be processed and acquired, avoiding motion blur artifacts and providing fine-grained temporal information. However, so far only a limited amount of datasets have been made freely accessible [22,36,47,75]. Despite the field is actively working towards increasing their availability, as testified by the recent release of event-based versions of ImageNet [54,63], relatively few datasets for human activity recognition are currently available.…”

Neuromorphic Engineering Needs Closed-Loop Benchmarks

“…Neuromorphic engineering has individually demonstrated many significant and valuable concepts, evidenced by dedicated large-scale neuromorphic processors (Davies et al, 2018 ), power-efficient analogue neuron circuits (Chicca et al, 2014 ; Moradi et al, 2018 ), on-chip and local unsupervised learning circuitry (Qiao et al, 2015 ), scalable parallel message-passing architectures (Furber, 2016 ), and retina-inspired and compressed visual sensing (Lichtsteiner et al, 2008 ). There are also active research and commercialisation efforts in applications of this research, including in Event-based Space Situational Awareness (Cohen et al, 2019 ), autonomous vehicle sensors (Perot et al, 2020 ; Gehrig et al, 2021 ), and for home security monitoring (Park et al, 2019 ; Samsung, 2020 ). However, the field struggles to integrate, build upon, and convey these successes to the wider engineering and scientific community.…”

“…Neuromorphic engineering has naturally followed a similar trajectory, both through the conversion of existing datasets to a neuromorphic format (Orchard et al, 2015a ) and through the collection and creation of new datasets (Perot et al, 2020 ; Gehrig et al, 2021 ). The growth of neuromorphic computing has further driven the need for suitable neuromorphic benchmarks to showcase the utility of its approaches to artificial intelligence.…”