Unsupervised learning of hierarchical representations with convolutional deep belief networks

Lee, Honglak; Grosse, Roger; Ranganath, Rajesh; Ng, Andrew Y.

doi:10.1145/2001269.2001295

Cited by 677 publications

(656 citation statements)

References 27 publications

Supporting

Mentioning

647

Contrasting

Unclassified

Order By: Relevance

“…The Convolutional DBN (Lee et al, 2009a) (with a probabilistic variant of MP, Sec. 5.11) combines ideas from CNNs and DBNs, and was successfully applied to audio classification (Lee et al, 2009b).…”

Section: : First Official Competitions Won By Rnns and With Mpcnnsmentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex Complete BIBTEX file (888 kB): http://www.idsia.ch/˜juergen/deep.bib Preface This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond, sometimes using "local search" to follow citations of citations backwards in time. Since not all DL publications properly acknowledge earlier relevant work, additional global search strategies were employed, aided by consulting numerous neural network experts. As a result, the present preprint mostly consists of references. Nevertheless, through an expert selection bias I may have missed important work. A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century. For these reasons, this work should be viewed as merely a snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send corrections and suggestions to juergen@idsia.ch.

show abstract

Section: : First Official Competitions Won By Rnns and With Mpcnnsmentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

show abstract

“…), future work will investigate the use of convolutional deep-belief networks (e.g., Lee et al 2011a) to boost recognition accuracy by learning the localized correlations in the observed spectra.…”

Section: Discussionmentioning

confidence: 99%

“…Figure 1 shows a schematic of the the deep-belief network. The multi-layer DBN can be constructed from several Restricted Boltzmann Machines (Freund & Haussler 1992;Bishop 2006;Le Roux & Bengio 2008;Bengio 2009Bengio , 2012Lee et al 2011a;Hinton 2012;Montavon et al 2012;Fischer & Igel 2014) with the addition of a logistic regression layer at the top of the network. The RBM is a two-layer neural network able to learn the underlying probability distribution over its set of input values.…”

Section: Robertmentioning

confidence: 99%

Dreaming of Atmospheres

Waldmann

2016

ApJ

View full text Add to dashboard Cite

Here, we introduce the RobERt(Robotic Exoplanet Recognition) algorithm for the classification of exoplanetary emission spectra. Spectral retrieval of exoplanetary atmospheres frequently requires the preselection of molecular/ atomic opacities to be defined by the user. In the era of open-source, automated, and self-sufficient retrieval algorithms, manual input should be avoided. User dependent input could, in worst-case scenarios, lead to incomplete models and biases in the retrieval. The RobERtalgorithm is based on deep-belief neural (DBN) networks trained to accurately recognize molecular signatures for a wide range of planets, atmospheric thermal profiles, and compositions. Reconstructions of the learned features, also referred to as the "dreams" of the network, indicate good convergence and an accurate representation of molecular features in the DBN. Using these deep neural networks, we work toward retrieval algorithms that themselves understand the nature of the observed spectra, are able to learn from current and past data, and make sensible qualitative preselections of atmospheric opacities to be used for the quantitative stage of the retrieval process.

show abstract

“…Focusing on these, there are various hierarchically organized models proposed, taking inspiration from different fields of science. There are nature inspired designs [3,4], there are models using grammars [5][6][7], there are very successful approaches using neural networks [8,9]. Learning strategies of such structures are similarly diverse, ranging from semi-automatic methods when the structure is given by human and only its parameters are learned [5] over sophisticated supervised/unsupervised methods of deep learning [10,9,8].…”

Section: Introductionmentioning

confidence: 99%

“…Noticeable difference from the deep learning is that unlike learning deep belief [10] or convolutional neural network [8] this method gives an explicit structure model similar to image grammars and requires less hyper-parameter/design choices -there is actually just one hyper-parameter that needs to be set and that is the maximal allowed portion of non-modelled data. Furthermore, its individual learned compositions can be used as features in more sophisticated classification framework such as SVM or AdaBoost.…”

Section: Introductionmentioning

confidence: 99%

A Simple Stochastic Algorithm for Structural Features Learning

Mačák

2015

Computer Vision - ACCV 2014 Workshops

View full text Add to dashboard Cite

Abstract. A conceptually very simple unsupervised algorithm for learning structure in the form of a hierarchical probabilistic model is described in this paper. The proposed probabilistic model can easily work with any type of image primitives such as edge segments, non-max-suppressed filter set responses, texels, distinct image regions, SIFT features, etc., and is even capable of modelling non-rigid and/or visually variable objects. The model has recursive form and consists of sets of simple and gradually growing sub-models that are shared and learned individually in layers. The proposed probabilistic framework enables to exactly compute the probability of presence of a certain model, regardless on which layer it actually is. All these learned models constitute a rich set of independent structure elements of variable complexity that can be used as features in various recognition tasks.

show abstract

Unsupervised learning of hierarchical representations with convolutional deep belief networks

Cited by 677 publications

References 27 publications

Deep learning in neural networks: An overview

Deep learning in neural networks: An overview

Dreaming of Atmospheres

A Simple Stochastic Algorithm for Structural Features Learning

Contact Info

Product

Resources

About