The "Wake-Sleep" Algorithm for Unsupervised Neural Networks

Hinton, Geoffrey E.; Dayan, Peter; Frey, Brendan J.; Neal, Radford M.

doi:10.1126/science.7761831

Cited by 827 publications

(620 citation statements)

References 6 publications

Supporting

Mentioning

600

Contrasting

Order By: Relevance

“…The target entering the opposite net slowly "cools down" to become the input. The Helmholtz Machine (Dayan et al, 1995;Dayan and Hinton, 1996) may be viewed as an unsupervised (Sec. 5.6.4) variant thereof (Peter Dayan, personal communication, 1994).…”

Section: Ideas For Dealing With Long Time Lags and Deep Capsmentioning

confidence: 99%

“…Many UL methods are designed to maximize entropy-related, information-theoretic (Boltzmann, 1909;Shannon, 1948;Kullback and Leibler, 1951) objectives (e.g., Linsker, 1988;Barlow et al, 1989;MacKay and Miller, 1990;Plumbley, 1991;Schmidhuber, 1992b,c;Schraudolph and Sejnowski, 1993;Redlich, 1993;Zemel, 1993;Zemel and Hinton, 1994;Field, 1994;Hinton et al, 1995;Dayan and Zemel, 1995;Amari et al, 1996;Deco and Parra, 1997).…”

Section: Potential Benefits Of Ul For Slmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex Complete BIBTEX file (888 kB): http://www.idsia.ch/˜juergen/deep.bib Preface This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond, sometimes using "local search" to follow citations of citations backwards in time. Since not all DL publications properly acknowledge earlier relevant work, additional global search strategies were employed, aided by consulting numerous neural network experts. As a result, the present preprint mostly consists of references. Nevertheless, through an expert selection bias I may have missed important work. A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century. For these reasons, this work should be viewed as merely a snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send corrections and suggestions to juergen@idsia.ch.

show abstract

Section: Ideas For Dealing With Long Time Lags and Deep Capsmentioning

confidence: 99%

Section: Potential Benefits Of Ul For Slmentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

show abstract

“…factor and cluster analysis) and those motivated by Bayesian inference and learning (e.g. Dayan et al, 1995;Hinton, Dayan, Frey, & Neal, 1995). The goal of generative models is "to learn representations that are economical to describe but allow the input to be reconstructed accurately" .…”

Section: The Nature Of Inputs Causes and Representationsmentioning

confidence: 99%

Learning and inference in the brain

2003

View full text Add to dashboard Cite

“…factor and cluster analysis) and those motivated by Bayesian inference and learning (e.g. Dayan et al, 1995;Hinton et al, 1995). Indeed many of the algorithms discussed under the heading of information theory can be formulated as generative models.…”

Section: Generative Models and Representational Learningmentioning

confidence: 99%

Functional integration and inference in the brain

Friston

2002

Progress in Neurobiology

271

201

View full text Add to dashboard Cite

Self-supervised models of how the brain represents and categorises the causes of its sensory input can be divided into two classes: those that minimise the mutual information (i.e. redundancy) among evoked responses and those that minimise the prediction error. Although these models have similar goals, the way they are attained, and the functional architectures employed, can be fundamentally different. This review describes the two classes of models and their implications for the functional anatomy of sensory cortical hierarchies in the brain. We then consider how empirical evidence can be used to disambiguate between architectures that are sufficient for perceptual learning and synthesis.Most models of representational learning require prior assumptions about the distribution of sensory causes. Using the notion of empirical Bayes, we show that these assumptions are not necessary and that priors can be learned in a hierarchical context. Furthermore, we try to show that learning can be implemented in a biologically plausible way. The main point made in this review is that backward connections, mediating internal or generative models of how sensory inputs are caused, are essential if the process generating inputs cannot be inverted. Because these processes are dynamical in nature, sensory inputs correspond to a non-invertible nonlinear convolution of causes. This enforces an explicit parameterisation of generative models (i.e. backward connections) to enable approximate recognition and suggests that feedforward architectures, on their own, are not sufficient. Moreover, nonlinearities in generative models, that induce a dependence on backward connections, require these connections to be modulatory; so that estimated causes in higher cortical levels can interact to predict responses in lower levels. This is important in relation to functional asymmetries in forward and backward connections that have been demonstrated empirically.To ascertain whether backward influences are expressed functionally requires measurements of functional integration among brain systems. This review summarises approaches to integration in terms of effective connectivity and proceeds to address the question posed by the theoretical considerations above. In short, it will be shown that functional neuroimaging can be used to test for interactions between bottom-up and top-down inputs to an area. The conclusion of these studies points toward the prevalence of top-down influences and the plausibility of generative models of sensory brain function.

show abstract

The "Wake-Sleep" Algorithm for Unsupervised Neural Networks

Cited by 827 publications

References 6 publications

Deep learning in neural networks: An overview

Deep learning in neural networks: An overview

Learning and inference in the brain

Functional integration and inference in the brain

Contact Info

Product

Resources

About