FastHebb: Scaling Hebbian Training of Deep Neural Networks to ImageNet Level

Hebbian semi-supervised learning in a sample efficiency setting

“…Another disadvantage of biologically based models is related to the computational cost of simulating certain synaptic dynamics or unfolding the temporal evolution of complex neural circuitry. A recent work [137] addresses in part this problem by leveraging GPU parallelization more carefully.…”

“…Furthermore, Hebbian learning was successfully used to retrain the higher layers of a pre-trained network, achieving results comparable to backprop, but requiring fewer training epochs, thus suggesting potential applications in the context of transfer learning (see also [32,156,157]). Some contributions [132,137] showed promising results of unsupervised Hebbian algorithms for semi-supervised network training, in learning scenarios with scarce data availability, achieving superior results compared to other backprop-based unsupervised methods for semi-supervised training such as Variational Auto-Encoders (VAE) [118]. In further developments [132,137], a more efficient formulation of Hebbian learning was also proposed, that enabled the scaling up of experiments to complex image recognition datasets, such as ImageNet [56], large-scale image retrieval, and complex network architectures, improving training speed up to a factor of 50.…”

“…Some contributions [132,137] showed promising results of unsupervised Hebbian algorithms for semi-supervised network training, in learning scenarios with scarce data availability, achieving superior results compared to other backprop-based unsupervised methods for semi-supervised training such as Variational Auto-Encoders (VAE) [118]. In further developments [132,137], a more efficient formulation of Hebbian learning was also proposed, that enabled the scaling up of experiments to complex image recognition datasets, such as ImageNet [56], large-scale image retrieval, and complex network architectures, improving training speed up to a factor of 50. The solution, named FastHebb, leveraged some observations that allowed us to rewrite Hebbian update equations in terms of matrix multiplications, to better exploit GPU acceleration.…”

“…Therefore, researchers took again inspiration from biology, in order to find new learning solutions as alternatives to backprop. The goal was not only to address the problem of SNN training [33,148], but also to discover novel approaches to the learning problem [77,139,182], and possibly more data efficient strategies [69,90,[105][106][107]110]. In fact, another limitation of current DL solutions is the requirement of Fig.…”

Lightweight and Elegant Data Reduction Strategies for Training Acceleration of Convolutional Neural Networks

“…Hebbian winner-take-all models are deemed feasible for neuromorphic computing in light of their characteristic of local and unsupervised weight update rule [55,56]. In contrast to previously mentioned approaches, the Hebbian learning method has demonstrated reasonable levels of convergence when used for training on large datasets such as ImageNet [57,58], while also exhibiting training time comparable to backpropagation [59]. Despite the progress made in the field, opportunities remain for improvement.…”