Biologically plausible deep learning — But how far can we go with shallow networks?

Abstract

A B S T R A C TTraining deep neural networks with the error backpropagation algorithm is considered implausible from a biological perspective. Numerous recent publications suggest elaborate models for biologically plausible variants of deep learning, typically defining success as reaching around 98% test accuracy on the MNIST data set. Here, we investigate how far we can go on digit (MNIST) and object (CIFAR10) classification with biologically plausible, local learning rules in a network with one hidden layer and a single readout layer. The hidden layer weights are either fixed (random or random Gabor filters) or trained with unsupervised methods (Principal/Independent Component Analysis or Sparse Coding) that can be implemented by local learning rules. The readout layer is trained with a supervised, local learning rule. We first implement these models with rate neurons. This comparison reveals, first, that unsupervised learning does not lead to better performance than fixed random projections or Gabor filters for large hidden layers. Second, networks with localized receptive fields perform significantly better than networks with all-to-all connectivity and can reach backpropagation performance on MNIST. We then implement two of the networks -fixed, localized, random & random Gabor filters in the hidden layer -with spiking leaky integrate-and-fire neurons and spike timing dependent plasticity to train the readout layer. These spiking models achieve > 98.2% test accuracy on MNIST, which is close to the performance of rate networks with one hidden layer trained with backpropagation. The performance of our shallow network models is comparable to most current biologically plausible models of deep learning. Furthermore, our results with a shallow spiking network provide an important reference and suggest the use of datasets other than MNIST for testing the performance of future models of biologically plausible deep learning.

Contrasting action and posture coding with hierarchical deep neural network models of proprioception

“…Neural responses have been found to be different for active and passive movements both in cortex, and are already different even at the level of spindles due to the gamma-fusimotor modulation (3,(57)(58)(59). We believe that exploring different objective functions (tasks), input types, learning rules (43) and architectural constraints will be very fruitful to elucidate proprioception with important applications for neural prostheses.…”

Social Determinants in Machine Learning Cardiovascular Disease Prediction Models: A Systematic Review

“…The authors refer to these studies collectively as Deep Learning/Neural Networks. 37 The publication of machine learning algorithms for CVD prediction has been increasing quickly since 2015, with the wide application of Neural Networks and Random Forest (Appendix Figure 1, available online), likely owing to the availability of software for ease of implementation and the availability of computing power resources for these algorithms that may otherwise take long compute times. Of the 21 studies including multiple algorithms, Random Forest (6 studies) and Support Vector Machine (4 studies) were most frequently reported as the best performing algorithms.…”

Machine Learning for Integrating Social Determinants in Cardiovascular Disease Prediction Models: A Systematic Review

“…Here we refer to these studies collectively as “neural networks” (NN) as deep learning typically refers to an neural network with multiple layers. 45 Of the 42 studies including multiple machine learning algorithms, random forest (9 studies) and neural network (9 studies) were most frequently reported as the best performing machine learning algorithms. For most commonly studied CVD outcomes, random forest was frequently reported to have the best prediction for stroke while support vector machine (SVM) performed best for coronary artery disease.…”