Exploring Temporal Information Dynamics in Spiking Neural Networks

Abstract

Most existing Spiking Neural Network (SNN) works state that SNNs may utilize temporal information dynamics of spikes. However, an explicit analysis of temporal information dynamics is still missing. In this paper, we ask several important questions for providing a fundamental understanding of SNNs: What are temporal information dynamics inside SNNs? How can we measure the temporal information dynamics? How do the temporal information dynamics affect the overall learning performance? To answer these questions, we estimate the Fisher Information of the weights to measure the distribution of temporal information during training in an empirical manner. Surprisingly, as training goes on, Fisher information starts to concentrate in the early timesteps. After training, we observe that information becomes highly concentrated in earlier few timesteps, a phenomenon we refer to as temporal information concentration. We observe that the temporal information concentration phenomenon is a common learning feature of SNNs by conducting extensive experiments on various configurations such as architecture, dataset, optimization strategy, time constant, and timesteps. Furthermore, to reveal how temporal information concentration affects the performance of SNNs, we design a loss function to change the trend of temporal information. We find that temporal information concentration is crucial to building a robust SNN but has little effect on classification accuracy. Finally, we propose an efficient iterative pruning method based on our observation on temporal information concentration. Code is available at https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.

Efficient Processing of Spiking Neural Networks via Task Specialization

“…Other general approaches to improve the processing efficiency of SNNs: A group of work re-used the network compression techniques via pruning and quantization for SNNs [14], [15], [16]. These techniques reduce the model size by reducing the neurons count (e.g., eliminating neurons with low firing rates), reducing the synaptic connectivity, or reducing the bit size of the network's parameters.…”

Shrinking Your TimeStep: Towards Low-Latency Neuromorphic Object Recognition with Spiking Neural Networks

“…Although this sample-wise timestep potentially limits parallel inference (batch size of 1 in (Li, Jones, and Furber 2023)), it demonstrates the potential of SNNs at very low latencies. Another work (Kim et al 2022) pointed out the phenomenon of temporal information concentration: during the training of SNNs, the effective information is gradually aggregated to the earlier timestep. The above work reveals the high temporal redundancy of existing SNNs, which raises a question: is it possible to reduce redundant timesteps in SNNs without sacrificing performance and parallelism?…”