Modeling and decoding event-based sensor lightning response

Abstract

Neuromorphic cameras, or Event-based Vision Sensors (EVS), operate in a fundamentally different way than conventional frame-based cameras. Their unique operational paradigm results in a sparse stream of high temporal resolution output events which encode pixel-level brightness changes with low-latency and wide dynamic range. Recently, interest has grown in exploiting these capabilities for scientific studies; however, accurately reconstructing signals from the output event stream presents a challenge due to physical limitations of the analog circuits that implement logarithmic change detection. In this paper, we present simultaneous recordings of lightning strikes using both an event camera and frame-based high-speed camera. To our knowledge, this is the first side-by-side recording using these two sensor types in a real-world scene with challenging dynamics that include very fast and bright illumination changes. Our goal in this work is to accurately map the illumination to EVS output in order to better inform modeling and reconstruction of events from a real-scene. We first combine lab measurements of key performance metrics to inform an existing pixel model. We then use the high-speed frames as signal ground truth to simulate an event stream and refine parameter estimates to optimally match the event-based sensor response for several dozen pixels representing different regions of the scene. These results will be used to predict sensor response and develop methods to more precisely reconstruct lightning and sprite signals for Falcon ODIN, our upcoming International Space Station neuromorphic sensing mission.

Sprite Durations Measured With a Neuromorphic Sensor

“…In a similar fashion to high-speed cameras, in dim observational conditions neuromorphic sensors begin to face physical limitations due to decreased photocurrent on the receptor. As discussed by Hu et al (2021) and McReynolds et al (2023), an individual pixel in a neuromorphic sensor is in great part an analog circuit, with its bandwidth proportional to the photocurrent, that is, proportional to the light incident upon the sensor. For dim sources, the bandwidth may be substantially reduced to a point where the photoreceptor can no longer follow the fluctuations in the source luminosity.…”

“…Each pixel of the sensor operates independently and only reports changes in brightness which surpass a preset threshold, greatly reducing the required data rate by eliminating redundant information. These systems also carry the benefit of a higher dynamic range and a higher maximum "equivalent frame rate" than cameras of comparable cost (Holesovsky et al, 2021;McReynolds et al, 2023;Scheerlinck, 2021). These proposed higher speeds and significantly lower data rates in principle have a lot of potential for sprite observations.…”

“…However, the time dynamics of streamer motion is right at the limits of neuromorphic sensors. As discussed byMcReynolds et al (2023) andHu et al (2021), in low-light conditions the effective temporal resolution of a neuromorphic sensor may not surpass 1 ms. Time resolution may even be worse for very dim sources (such as sprites, see below) because of the direct relationship between light intensity incident upon the photoreceptor and the effective response time (bandwidth).…”