Align, then memorise: the dynamics of learning with feedback alignment

Refinetti, Maria; d’Ascoli, Stéphane; Ohana, Ruben; Goldt, Sebastian

doi:10.48550/arxiv.2011.12428

Cited by 4 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While RBP addresses the Symmetry problem, by itself it does not address the other problems. Furthermore, while RBP simulations have succeeded on the MNIST and CIFAR benchmark data sets, it has been noted that RBP algorithms do not work well with convolutional layers [Bartunov et al, 2018, Moskovitz et al, 2018, Refinetti et al, 2020. A few methods have been proposed to address this apparent weakness of RBP algorithms, however, most of them are not biologically plausible.…”

Section: Random Backpropagationmentioning

confidence: 99%

“…After building the twin Tourbillon architectures and training their circular autoencoders by recirculation, we apply fine-tuning as described in Section 5.1 to perform random backpropagation using the channel provided by the decoders' connections. Empirically, we found that it is necessary to have different learning rate schedules for different layers during the fine-tuning phase to ensure a form of asynchronous learning among layers(see also Refinetti et al [2020]). A summary of the results comparing the various approaches is given in Table 1 (first row), and shows that the performance of the Tourbillon twin approach is comparable to backpropagation.…”

Section: Tourbillonizationmentioning

confidence: 99%

See 1 more Smart Citation

Tourbillon: a Physically Plausible Neural Architecture

Tavakoli,

Sadowski,

Baldi

2021

Preprint

View full text Add to dashboard Cite

In a physical neural system, backpropagation is faced with a number of obstacles including: the need for labeled data, the violation of the locality learning principle, the need for symmetric connections, and the lack of modularity. Tourbillon is a new architecture that addresses all these limitations. At its core, it consists of a stack of circular autoencoders followed by an output layer. The circular autoencoders are trained in self-supervised mode by recirculation algorithms and the top layer in supervised mode by stochastic gradient descent, with the option of propagating error information through the entire stack using non-symmetric connections. While the Tourbillon architecture is meant primarily to address physical constraints, and not to improve current engineering applications of deep learning, we demonstrate its viability on standard benchmark datasets including MNIST, Fashion MNIST, and CIFAR10. We show that Tourbillon can achieve comparable performance to models trained with backpropagation and outperform models that are trained with other physically plausible algorithms, such as feedback alignment.Preprint. Under review.

show abstract

Section: Random Backpropagationmentioning

confidence: 99%

Section: Tourbillonizationmentioning

confidence: 99%

Tourbillon: a Physically Plausible Neural Architecture

Tavakoli,

Sadowski,

Baldi

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Learning with DFA is enabled by an alignment process, wherein the forward weights learn a configuration enabling DFA to approximate BP updates [42].…”

Section: Our Defensementioning

confidence: 99%

ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients

Cappelli,

Launay,

Meunier

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Robustness to adversarial attacks is typically obtained through expensive adversarial training with Projected Gradient Descent. Here we introduce ROPUST, a remarkably simple and efficient method to leverage robust pre-trained models and further increase their robustness, at no cost in natural accuracy. Our technique relies on the use of an Optical Processing Unit (OPU), a photonic co-processor, and a fine-tuning step performed with Direct Feedback Alignment, a synthetic gradient training scheme. We test our method on nine different models against four attacks in RobustBench, consistently improving over state-of-the-art performance. We perform an ablation study on the single components of our defense, showing that robustness arises from parameter obfuscation and the alternative training method. We also introduce phase retrieval attacks, specifically designed to increase the threat level of attackers against our own defense. We show that even with state-of-the-art phase retrieval techniques, ROPUST remains an effective defense.

show abstract

“…For ease of notation, we introduce it for fully connected networks but it generalizes to convolutional networks, transformers and other architectures [16]. It has been theoretically studied in [20,26]. Note that in the following, we incorporate the bias terms in the weight matrices.…”

Section: Learning With Direct Feedback Alignment (Dfa)mentioning

confidence: 99%

Photonic Differential Privacy with Direct Feedback Alignment

Ohana¹,

Ruiz²,

Launay³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Optical Processing Units (OPUs) -low-power photonic chips dedicated to large scale random projections -have been used in previous work to train deep neural networks using Direct Feedback Alignment (DFA), an effective alternative to backpropagation. Here, we demonstrate how to leverage the intrinsic noise of optical random projections to build a differentially private DFA mechanism, making OPUs a solution of choice to provide a private-by-design training. We provide a theoretical analysis of our adaptive privacy mechanism, carefully measuring how the noise of optical random projections propagates in the process and gives rise to provable Differential Privacy. Finally, we conduct experiments demonstrating the ability of our learning procedure to achieve solid end-task performance.

show abstract

Align, then memorise: the dynamics of learning with feedback alignment

Cited by 4 publications

References 0 publications

Tourbillon: a Physically Plausible Neural Architecture

Tourbillon: a Physically Plausible Neural Architecture

ROPUST: Improving Robustness through Fine-tuning with Photonic Processors and Synthetic Gradients

Photonic Differential Privacy with Direct Feedback Alignment

Contact Info

Product

Resources

About