Back-Propagation Learning in Deep Spike-By-Spike Networks

Abstract

Artificial neural networks (ANNs) are important building blocks in technical applications. They rely on noiseless continuous signals in stark contrast to the discrete action potentials stochastically exchanged among the neurons in real brains. We propose to bridge this gap with Spike-by-Spike (SbS) networks which represent a compromise between non-spiking and spiking versions of generative models. What is missing, however, are algorithms for finding weight sets that would optimize the output performances of deep SbS networks with many layers. Here, a learning rule for feed-forward SbS networks is derived. The properties of this approach are investigated and its functionality is demonstrated by simulations. In particular, a Deep Convolutional SbS network for classifying handwritten digits achieves a classification performance of roughly 99.3% on the MNIST test data when the learning rule is applied together with an optimizer. Thereby it approaches the benchmark results of ANNs without extensive parameter optimization. We envision this learning rule for SBS networks to provide a new basis for research in neuroscience and for technical applications, especially when they become implemented on specialized computational hardware.

Learnability of the Boolean Innerproduct in Deep Neural Networks

“…Since the Boolean inner product is based on the parity function, we first looked how parity can be directly learned. The parity function is notorious for being very hard to learn with back-propagation [ 39 , 40 , 41 , 42 ], especially when somehow minimal networks are used. We then investigated the learnability of the Boolean inner product with the standard solution architectures, the shallow DNF architecture and the deep exact architecture.…”

Competitive performance and superior noise robustness of a non-negative deep convolutional spiking network

“…The network architecture. In the case of NNMF and SbS networks, the convolution layer (conv), the full layer, and the output layer are constructed from IPs with their according dynamic (equation 1 and 3) on their latent variables (see [4] for more details). For the CNN, the convolution layers, and the full layer are followed by ReLU layers.…”

“…For the CNN, the convolution layers, and the full layer are followed by ReLU layers. Also the pooling layers are max pooling layers instead of average pooing layers used for NNMF and SbS For NNMF and SbS layers we introduced the so called inference populations (IPs) [4]. Every IPs operates independent of the other IPs in a layer.…”

“…In this paper, we compare the performance of three different model classes for deep learning: standard deep convolutional neural networks (CNNs) [2], deep networks based on non-negative matrix factorization (NNMF) [5] [6], and a recent deep spiking neuronal network model called spike-by-spike (SbS) [3] [4]. Despite the differences in their underlying computational mechanisms we find that all three model classes achieved equivalent performance on a variety of tasks.…”

Current Characteristics Estimation of Si PV Modules Based on Artificial Neural Network Modeling

“…A combination of MLP and the gradient descent method results in a very effective algorithm, known as the back propagation (BP) algorithm [25,26,27,28,29]. The main idea of the gradient descent method is to make the weight of each node move into the negative direction of the loss function gradient and make the network adjust the weight value of each node by itself.…”