The HSIC Bottleneck: Deep Learning without Back-Propagation

Ma, Wan-Duo Kurt; Lewis, J. P.; Kleijn, W. Bastiaan

doi:10.48550/arxiv.1908.01580

Cited by 4 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…8). Like Ma, Lewis, and Kleijn (2019), we use a linear readout layer trained for 1000 epochs with gradient descent to map between the HSIC-learned output and the label encoding. This is only required for our experiments.…”

Section: Methodsmentioning

confidence: 99%

“…Though computing the mutual information of two random variables requires knowledge of their distributions, Ma, Lewis, and Kleijn (2019) propose using the Hilbert-Schmidt Independence Criterion (HSIC) as a proxy for mutual information. Given a finite number of samples, N , a statistical estimator for the HSIC (Gretton et al, 2005) is…”

Section: The Information Bottleneckmentioning

confidence: 99%

“…3 and 4 define the centered and uncentered kernel matrices, respectively. Using these definitions Ma, Lewis, and Kleijn (2019) define the HSIC objective -a loss function for training feedforward neural networks by balancing the IB at each layer. Consider a feedforward neural network with L layers where the output of layer is…”

Section: The Information Bottleneckmentioning

confidence: 99%

“…In this work, we rely on recent advances in deep learning that train feedforward networks by balancing the information bottleneck (Ma, Lewis, and Kleijn, 2019). Unlike back-propagation, where an error signal computed at the end of the network is propagated to the front (see Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Recent progress (Lillicrap et al, 2014(Lillicrap et al, , 2020 addresses part of this question, e.g., the weight transport problem, but a complete solution remains intangible. In contrast, novel learning rules (Ma, Lewis, and Kleijn, 2019;Pogodin and Latham, 2020) based on the information bottleneck (IB) train each layer of a network independently, circumventing the need to propagate errors across layers. Instead, propagation is implicit due the layers' feedforward connectivity.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Information Bottleneck-Based Hebbian Learning Rule Naturally Ties Working Memory and Synaptic Updates

Daruwalla¹,

Lipasti²

2021

Preprint

View full text Add to dashboard Cite

While deep feedforward neural networks are effective models for a wide array of problems, back-propagation, which makes training such networks possible, is biologically implausible. Neuroscientists are uncertain about how the brain would propagate a precise error signal backward through a network of neurons. Recent progress (Lillicrap et al., 2014(Lillicrap et al., , 2020 addresses part of this question, e.g., the weight transport problem, but a complete solution remains intangible. In contrast, novel learning rules (Ma, Lewis, and Kleijn, 2019;Pogodin and Latham, 2020) based on the information bottleneck (IB) train each layer of a network independently, circumventing the need to propagate errors across layers. Instead, propagation is implicit due the layers' feedforward connectivity. These rules take the form of a three-factor Hebbian update -a global error signal modulates local synaptic updates within each layer. Unfortunately, the global signal for a given layer requires processing multiple samples concurrently, and the brain only sees a single sample at a time. Prior work limits the rule to an approximated two-point update to preserve biological plausibility. Our findings show that this restriction negatively impacts the IB estimate and convergence. Instead, we propose a new three-factor update rule where the global signal correctly captures information across samples via an auxiliary reservoir network. The auxiliary network can be trained a priori independently of the dataset being used with the primary network. We demonstrate comparable performance to baselines on image classification tasks. Interestingly, unlike back-propagation-like schemes where there is no link between learning and memory, our rule presents a direct connection between working memory and synaptic updates. To the best of our knowledge, this is the first rule to make this link explicit. We explore these implications in initial experiments examining the effect of memory capacity on learning performance. Moving forward, this work suggests an alternate view of learning where each layer balances memory-informed

show abstract

Section: Methodsmentioning

confidence: 99%

Section: The Information Bottleneckmentioning

confidence: 99%

Section: The Information Bottleneckmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Information Bottleneck-Based Hebbian Learning Rule Naturally Ties Working Memory and Synaptic Updates

Daruwalla¹,

Lipasti²

2021

Preprint

View full text Add to dashboard Cite

show abstract

Information bottleneck-based Hebbian learning rule naturally ties working memory and synaptic updates

Daruwalla,

Lipasti

2024

Front. Comput. Neurosci.

View full text Add to dashboard Cite

Deep neural feedforward networks are effective models for a wide array of problems, but training and deploying such networks presents a significant energy cost. Spiking neural networks (SNNs), which are modeled after biologically realistic neurons, offer a potential solution when deployed correctly on neuromorphic computing hardware. Still, many applications train SNNs offline, and running network training directly on neuromorphic hardware is an ongoing research problem. The primary hurdle is that back-propagation, which makes training such artificial deep networks possible, is biologically implausible. Neuroscientists are uncertain about how the brain would propagate a precise error signal backward through a network of neurons. Recent progress addresses part of this question, e.g., the weight transport problem, but a complete solution remains intangible. In contrast, novel learning rules based on the information bottleneck (IB) train each layer of a network independently, circumventing the need to propagate errors across layers. Instead, propagation is implicit due the layers' feedforward connectivity. These rules take the form of a three-factor Hebbian update a global error signal modulates local synaptic updates within each layer. Unfortunately, the global signal for a given layer requires processing multiple samples concurrently, and the brain only sees a single sample at a time. We propose a new three-factor update rule where the global signal correctly captures information across samples via an auxiliary memory network. The auxiliary network can be trained a priori independently of the dataset being used with the primary network. We demonstrate comparable performance to baselines on image classification tasks. Interestingly, unlike back-propagation-like schemes where there is no link between learning and memory, our rule presents a direct connection between working memory and synaptic updates. To the best of our knowledge, this is the first rule to make this link explicit. We explore these implications in initial experiments examining the effect of memory capacity on learning performance. Moving forward, this work suggests an alternate view of learning where each layer balances memory-informed compression against task performance. This view naturally encompasses several key aspects of neural computation, including memory, efficiency, and locality.

show abstract

Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks

Pogodin,

Latham

2020

Preprint

View full text Add to dashboard Cite

The state-of-the art machine learning approach to training deep neural networks, backpropagation, is implausible for real neural networks: neurons need to know their outgoing weights; training alternates between a forward pass (computation) and a backward pass (learning); and the algorithm needs a large amount of labeled data. Biologically plausible approximations to backpropagation, such as feedback alignment, solve the weight transport problem, but not the other two. Thus, fully biologically plausible learning rules have so far remained elusive. Here we present a family of learning rules that does not suffer from any of these problems. It is motivated by the information bottleneck principle (extended with kernel methods), in which networks learn to squeeze as much information as possible out of the input without sacrificing prediction of the output. The resulting rules have a 3-factor Hebbian structure: they require pre-and post-synaptic firing rates and a global error signal -the third factor -that can be supplied by a neuromodulator. Moreover, they do not require precise labels; instead, they rely on the similarity between the desired outputs. They thus solve all three implausibility issues of backpropagation. Moreover, to obtain good performance on hard problems and retain biologically plausible learning rules, our rules need divisive normalization -a known feature of biological networks. Finally, simulations show that our rule performs nearly as well as backpropagation on image classification tasks.Preprint. Under review.

show abstract

The HSIC Bottleneck: Deep Learning without Back-Propagation

Cited by 4 publications

References 0 publications

Information Bottleneck-Based Hebbian Learning Rule Naturally Ties Working Memory and Synaptic Updates

Information Bottleneck-Based Hebbian Learning Rule Naturally Ties Working Memory and Synaptic Updates

Information bottleneck-based Hebbian learning rule naturally ties working memory and synaptic updates

Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks

Contact Info

Product

Resources

About