On Information and Sufficiency

Kullback, S.; Leibler, R. A.

doi:10.1214/aoms/1177729694

Cited by 16,159 publications

(9,504 citation statements)

References 0 publications

Supporting

Mentioning

8,962

Contrasting

Unclassified

Order By: Relevance

“…Many UL methods are designed to maximize entropy-related, information-theoretic (Boltzmann, 1909;Shannon, 1948;Kullback and Leibler, 1951) objectives (e.g., Linsker, 1988;Barlow et al, 1989;MacKay and Miller, 1990;Plumbley, 1991;Schmidhuber, 1992b,c;Schraudolph and Sejnowski, 1993;Redlich, 1993;Zemel, 1993;Zemel and Hinton, 1994;Field, 1994;Hinton et al, 1995;Dayan and Zemel, 1995;Amari et al, 1996;Deco and Parra, 1997).…”

Section: Potential Benefits Of Ul For Slmentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex Complete BIBTEX file (888 kB): http://www.idsia.ch/˜juergen/deep.bib Preface This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond, sometimes using "local search" to follow citations of citations backwards in time. Since not all DL publications properly acknowledge earlier relevant work, additional global search strategies were employed, aided by consulting numerous neural network experts. As a result, the present preprint mostly consists of references. Nevertheless, through an expert selection bias I may have missed important work. A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century. For these reasons, this work should be viewed as merely a snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send corrections and suggestions to juergen@idsia.ch.

show abstract

Section: Potential Benefits Of Ul For Slmentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

show abstract

“…Kullback-Liebler distance (Kullback & Liebler, 1951;Cover & Thomas, 1991) provides one means to measure the change from prior beliefs about the true category, C, to posterior beliefs after a particular question is answered:…”

Section: Kullback-liebler Distancementioning

confidence: 99%

Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.

Nelson

2005

Psychological Review

222

403

View full text Add to dashboard Cite

Several norms for how people should assess a question's usefulness have been proposed, notably Bayesian diagnosticity, information gain (mutual information), Kullback-Liebler distance, probability gain (error minimization), and impact (absolute change). Several probabilistic models of previous experiments on categorization, covariation assessment, medical diagnosis, and the selection task are shown to not discriminate among these norms as descriptive models of human intuitions and behavior. Computational optimization found situations in which information gain, probability gain, and impact strongly contradict Bayesian diagnosticity. In these situations, diagnosticity's claims are normatively inferior. Results of a new experiment strongly contradict the predictions of Bayesian diagnosticity. Normative theoretical concerns also argue against use of diagnosticity. It is concluded that Bayesian diagnosticity is normatively flawed and empirically unjustified.

show abstract

“…), i.e., which PB have the most informative contacts in terms of contact distribution, and (ii) by determining which PB is preferentially associated to a given PB. To deal with the first point, we have used the relative entropy or Kullback-Leibler asymmetric divergence measure [94].…”

Section: Analysis Of the Over-and Under-represented Contactsmentioning

confidence: 99%

Analysis of protein contacts into Protein Units

2009

View full text Add to dashboard Cite

Three-dimensional structures of proteins are the support of their biological functions.Their folds are maintained by inter-residue interactions which are one of the main focuses to understand the mechanisms of protein folding and stability. Furthermore, protein structures can be composed of single or multiple functional domains that can fold and function independently. Hence, dividing a protein into domains is useful for obtaining an accurate structure and function determination.In previous studies, we enlightened protein contact properties according to different

show abstract

On Information and Sufficiency

Cited by 16,159 publications

References 0 publications

Deep learning in neural networks: An overview

Deep learning in neural networks: An overview

Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.

Analysis of protein contacts into Protein Units

Contact Info

Product

Resources

About