Imaging from temporal data via spiking convolutional neural networks

Abstract

A new approach for imaging that is solely based on the time of flight of photons coming from the entire imaged scene, combined with a novel machine learning algorithm for image reconstruction: a spiking convolutional neural network (SCNN) named Spike-SPI (Spiking-Single Pixel Imager). The approach uses a single point detector and the corresponding time-counting electronics, which provide the arrival time of photons in the form of spikes distributed over time. This data is transformed into a temporal histogram containing the number of photons per arrival time. A SCNN that converts the 1D temporal histograms into a 3D image (2D image with depth map) by exploiting the feature extraction capabilities of convolutional neural networks (CNNs), the high dimensional compressed latent space representations of a variational encoder-decoder network structure, and the asynchronous processing capabilities of a spiking neural network (SNN). The performance of the proposed SCNN is analysed to demonstrate the state-of-the-art feature extraction capabilities of CNNs and the low latency asynchronous processing of SNNs that offer both higher throughput and higher accuracy in image reconstruction from the ToF data, when compared to standard ANNs. The results of Spike-SPI show an increase in spatial accuracy of 15% over then ANN, using the Intersection of Union (IoU) for the objects in the scene. While also delivering a 100% increase over then ANN in object reconstruction signal to noise ratio (RSNR) from ∼3dB to ∼6dB. These results are also consistent across a range of IRF (Instrument Response Functions) values and photo counts, highlighting the robust nature of the new network structure. Moreover, the asynchronous processing nature of the spiking neurons allow for a faster throughput and less computational overhead, benefiting from the operational sparsity in the single point sensor.

VCSEL-based photonic spiking neural networks for ultrafast detection and tracking

“…For example, software-implemented neuromorphic SNNs have been used to develop multiple layer hierarchical convolution operations to extract image features patterns in images for classification purposes [27]. Further, software-implemented SNNs, and the sparsity of their spiking representation, have also been shown to pair especially well with hardware sensors such as single photon detectors [28] and event-based cameras [29], providing image processing functionalities with greater reductions in computational processing. The realisation of SNNs with the photonic platform therefore holds exciting prospects for neuromorphic image processing systems with low computational requirements, light-speed operation, and potentially low power requirements.…”