Meta Learning Backpropagation And Improving It

Kirsch, Louis; Schmidhuber, Jürgen

doi:10.48550/arxiv.2012.14905

Cited by 6 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A similar direction has been taken by Kirsch et al 33 , where the neurons and synapses of a neural network are also generalized to higher dimension message-passing systems, but in their case each synapse is replaced by an recurrent neural network (RNN) with the same shared parameters. These RNN synapses are bi-directional and govern the flow of information across the network.…”

Section: Meta-learningmentioning

confidence: 94%

“…Since RNNs are general-purpose computers, they were able to demonstrate that the system can encode the gradient-based backpropagation algorithm by training the system to simply emulate backpropagation, rather than explicitly calculating gradients via hand-engineering. Of course, 55 and Kirsch et al 33 attempt to generalize the accepted notion of artificial neural networks, where each neuron can hold multiple states rather than a scalar value, and each synapse function bi-directionally to facilitate both learning and inference. In this figure, Kirsch et al 33 proposes using a identical recurrent neural network (RNN) (with different internal hidden states) to model each synapse, and show the network can be trained by simply running the RNN cells, without the use of backpropagation.…”

Section: Meta-learningmentioning

confidence: 99%

“…Figure 10. Recent work by Sandler et al55 and Kirsch et al33 attempt to generalize the accepted notion of artificial neural networks, where each neuron can hold multiple states rather than a scalar value, and each synapse function bi-directionally to facilitate both learning and inference. In this figure, Kirsch et al33 proposes using a identical recurrent neural network (RNN) (with different internal hidden states) to model each synapse, and show the network can be trained by simply running the RNN cells, without the use of backpropagation.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Collective Intelligence for Deep Learning: A Survey of Recent Developments

Ha¹,

Tang²

2021

Preprint

View full text Add to dashboard Cite

In the past decade, we have witnessed the rise of deep learning to dominate the field of artificial intelligence. Advances in artificial neural networks alongside corresponding advances in hardware accelerators with large memory capacity, together with the availability of large datasets enabled researchers and practitioners alike to train and deploy sophisticated neural network models that achieve state-of-the-art performance on tasks across several fields spanning computer vision, natural language processing, and reinforcement learning. However, as these neural networks become bigger, more complex, and more widely used, fundamental problems with current deep learning models become more apparent. State-of-the-art deep learning models are known to suffer from issues that range from poor robustness, inability to adapt to novel task settings, to requiring rigid and inflexible configuration assumptions. Ideas from collective intelligence, in particular concepts from complex systems such as self-organization, emergent behavior, swarm optimization, and cellular systems tend to produce solutions that are robust, adaptable, and have less rigid assumptions about the environment configuration. It is therefore natural to see these ideas incorporated into newer deep learning methods. In this review, we will provide a historical context of neural network research's involvement with complex systems, and highlight several active areas in modern deep learning research that incorporate the principles of collective intelligence to advance its current capabilities. To facilitate a bi-directional flow of ideas, we also discuss work that utilize modern deep learning models to help advance complex systems research. We hope this review can serve as a bridge between complex systems and deep learning communities to facilitate the cross pollination of ideas and foster new collaborations across disciplines.

show abstract

Section: Meta-learningmentioning

confidence: 94%

Section: Meta-learningmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Collective Intelligence for Deep Learning: A Survey of Recent Developments

Ha¹,

Tang²

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…A meta-learned policy that can adapt the weights of a neural network to its inputs during inference time have been proposed in fast weights [64,66], associative weights [2], hypernetworks [35], and Hebbian-learning [51,52] approaches. Recently works [45,62] combine ideas of self-organization with meta-learning RNNs, and have demonstrated that modular metalearning RNN systems not only can learn to perform SGD-like learning rules, but can also discover more general learning rules that transfer to classification tasks on unseen datasets.…”

Section: Related Workmentioning

confidence: 99%

The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning

Tang¹,

Ha²

2021

Preprint

View full text Add to dashboard Cite

In complex systems, we often observe complex global behavior emerge from a collection of agents interacting with each other in their environment, with each individual agent acting only on locally available information, without knowing the full picture. Such systems have inspired development of artificial intelligence algorithms in areas such as swarm optimization and cellular automata. Motivated by the emergence of collective behavior from complex cellular systems, we build systems that feed each sensory input from the environment into distinct, but identical neural networks, each with no fixed relationship with one another. We show that these sensory networks can be trained to integrate information received locally, and through communication via an attention mechanism, can collectively produce a globally coherent policy. Moreover, the system can still perform its task even if the ordering of its inputs is randomly permuted several times during an episode. These permutation invariant systems also display useful robustness and generalization properties that are broadly applicable. Interactive demo and videos of our results: https://attentionneuron.github.io/

show abstract

“…Meta-optimizers. Meta-optimizers [17,18,71,91,92] define a problem similar to our task, but where H D is an RNN-based model predicting the gradients ∇w, mimicking the behavior of iterative optimizers. Therefore, the objective of meta-optimizers may be phrased as learning to optimize as opposed to our learning to predict parameters.…”

Section: Related Workmentioning

confidence: 99%

Parameter Prediction for Unseen Deep Architectures

Knyazev¹,

Drozdzal²,

Taylor³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures -DEEPNETS-1M-and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.

show abstract

Meta Learning Backpropagation And Improving It

Cited by 6 publications

References 0 publications

Collective Intelligence for Deep Learning: A Survey of Recent Developments

Collective Intelligence for Deep Learning: A Survey of Recent Developments

The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning

Parameter Prediction for Unseen Deep Architectures

Contact Info

Product

Resources

About