Learning Temporal-Ordered Representation for Spike Streams Based on Discrete Wavelet Transforms

Abstract

Spike camera, a new type of neuromorphic visual sensor that imitates the sampling mechanism of the primate fovea, can capture photons and output 40000 Hz binary spike streams. Benefiting from the asynchronous sampling mechanism, the spike camera can record fast-moving objects and clear images can be recovered from the spike stream at any specified timestamps without motion blurring. Despite these, due to the dense time sequence information of the discrete spike stream, it is not easy to directly apply the existing algorithms of traditional cameras to the spike camera. Therefore, it is necessary and interesting to explore a universally effective representation of dense spike streams to better fit various network architectures. In this paper, we propose to mine temporal-robust features of spikes in time-frequency space with wavelet transforms. We present a novel Wavelet-Guided Spike Enhancing (WGSE) paradigm consisting of three consecutive steps: multi-level wavelet transform, CNN-based learnable module, and inverse wavelet transform. With the assistance of WGSE, the new streaming representation of spikes can be learned. We demonstrate the effectiveness of WGSE on two downstream tasks, achieving state-of-the-art performance on the image reconstruction task and getting considerable performance on semantic segmentation. Furthermore, We build a new spike-based synthesized dataset for semantic segmentation. Code and Datasets are available at https://github.com/Leozhangjiyuan/WGSE-SpikeCamera.

Transient Glimpses: Unveiling Occluded Backgrounds through the Spike Camera

“…These advantages have led to wide applications in various downstream tasks, such as optical flow estimation (Hu et al 2022;Zhao et al 2022;, object tracking (Zheng et al 2023a), and depth estimation (Zhang et al 2022a;Wang et al 2022). Among these tasks, the reconstruction task (Zhang et al 2023) serves as the fundamental basis. In the early stages, Zhu et al (Zhu et al 2019) propose to approximate the light intensity by statistically analyzing the spike stream.…”