Streaming parallel GPU acceleration of large-scale filter-based spiking neural networks

Abstract

The arrival of graphics processing (GPU) cards suitable for massively parallel computing promises a↵ordable large-scale neural network simulation previously only available at supercomputing facilities. While the raw numbers suggest that GPUs may outperform CPUs by at least an order of magnitude, the challenge is to develop fine-grained parallel algorithms to fully exploit the particulars of GPUs. Computation in a neural network is inherently parallel and thus a natural match for GPU architectures: given inputs, the internal state for each neuron can be updated in parallel. We show that for filter-based spiking neurons, like the Spike Response Model, the additive nature of membrane potential dynamics enables additional update parallelism. This also reduces the accumulation of numerical errors when using single precision computation, the native precision of GPUs. We further show that optimizing simulation algorithms and data structures to the GPU's architecture has a large pay-o↵: for example, matching iterative neural updating to the memory architecture of the GPU speeds up this simulation step by a factor of three to five. With such optimizations, we can simulate in better-than-realtime plausible spiking neural networks of up to 50,000 neurons, processing over 35 million spiking events per second.

Fast and Efficient Asynchronous Neural Computation with Adapting Spiking Neural Networks

“…This analysis ignores the fact that spikes in the ASNN (and SNN) are heavily localized to a subset of neurons: many neuron are silent while a few are active. Sparse and localised communication potentially offers a benefit to deep neural networks, as densely connected neural networks tend to be limited by the bandwidth required to read and write the appropriate weights from memory [21]. Thus reasoned, for an ASNN that incurs a 100ms delay to compete in terms of bandwidth used with an ANN, it can use at most a firing rate of 10Hz on average per neuron, since an ANN sampled with 10Hz would achieve the same worst case delay.…”

Dynamic parallelism for synaptic updating in GPU-accelerated spiking neural network simulations

“…Different parallelization strategies have been designed for GPUs (reviewed in [8]) with the aim to vectorize these calculations: across neurons [9], or across spikes and synapses [3]. However, these strategies all have in common that they run many obsolete operations in each time-step, as they typically include also the silent (non-spiking) neurons in the network.…”