Neural Networks and the Bias/Variance Dilemma

Geman, Stuart; Bienenstock, Elie; Doursat, René

doi:10.1162/neco.1992.4.1.1

Cited by 2,910 publications

(1,602 citation statements)

References 49 publications

Supporting

Mentioning

1,519

Contrasting

Unclassified

Order By: Relevance

“…Our arguments in favor of structured models build on claims made by previous researchers (Geman et al, 1992;Chomsky, 1986), and psychologists and AI researchers have been developing structured models for many years (Collins & Quillian, 1969;Davis, 1990;Lenat, 1995).…”

Section: Modeling Frameworkmentioning

confidence: 83%

Structured statistical models of inductive reasoning.

Kemp¹,

Tenenbaum²

2009

Psychological Review

222

282

View full text Add to dashboard Cite

Everyday inductive inferences are often guided by rich background knowledge. Formal models of induction should aim to incorporate this knowledge, and should explain how different kinds of knowledge lead to the distinctive patterns of reasoning found in different inductive contexts. We present a Bayesian framework that attempts to meet both goals and describe four applications of the framework: a taxonomic model, a spatial model, a threshold model, and a causal model. Each model makes probabilistic inferences about the extensions of novel properties, but the priors for the four models are defined over different kinds of structures that capture different relationships between the categories in a domain. Our framework therefore shows how statistical inference can operate over structured background knowledge, and we argue that this interaction between structure and statistics is critical for explaining the power and flexibility of human reasoning. 2 Structured statistical models of inductive reasoningHumans are adept at making inferences that take them beyond the limits of their direct experience. Even young children can learn the meaning of a novel word from a single labeled example (Heibeck & Markman, 1987), predict the trajectory of a moving object when it passes behind an occluder (Spelke, 1990), and choose a gait that allows them to walk over terrain they have never before encountered. Inferences like these may differ in many respects, but common to them all is the need to "go beyond the information given" (Bruner, 1973).Two different ways of going beyond the available information can be distinguished.Deductive inferences draw out conclusions that may have been previously unstated but were implicit in the data provided. Inductive inferences go beyond the available data in a more fundamental way, and arrive at conclusions that are likely but not certain given the available evidence. Both kinds of inferences are of psychological interest, but inductive inferences appear to play a more central role in everyday cognition. We have already seen examples related to language, vision, and motor control, and many other inductive problems have been described in the literature (Holland, Holyoak, Nisbett, & Thagard, 1986;Anderson, 1990). This paper describes a formal approach to inductive inference that should apply to many different problems, but we will focus on the problem of property induction (Sloman & Lagnado, 2005). In particular, we consider cases where one or more more categories in a domain are observed to have a novel property, and the inductive task is to predict how the property is distributed over the remaining categories in the domain. For instance, given that bears have sesamoid bones, which species is more likely to share this property: moose or salmon (Rips, 1975; Osherson, Smith, Wilkie, Lopez, & Shafir, 1990)? Moose may seem like the better choice since they are more similar biologically to bears, but different properties can lead to different patterns of inference. For example, given that a c...

show abstract

Section: Modeling Frameworkmentioning

confidence: 83%

Structured statistical models of inductive reasoning.

Kemp¹,

Tenenbaum²

2009

Psychological Review

222

282

View full text Add to dashboard Cite

show abstract

“…4.4) with high generalization capability. Most approaches address the bias/variance dilemma (Geman et al, 1992) through strong prior assumptions. For example, weight decay (Hanson and Pratt, 1989;Weigend et al, 1991;Krogh and Hertz, 1992) encourages near-zero weights, by penalizing large weights.…”

Section: Better Bp Through Advanced Gradient Descent (Compare Sec 524)mentioning

confidence: 99%

Deep learning in neural networks: An overview

2015

View full text Add to dashboard Cite

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.LATEX source: http://www.idsia.ch/˜juergen/DeepLearning8Oct2014.tex Complete BIBTEX file (888 kB): http://www.idsia.ch/˜juergen/deep.bib Preface This is the preprint of an invited Deep Learning (DL) overview. One of its goals is to assign credit to those who contributed to the present state of the art. I acknowledge the limitations of attempting to achieve this goal. The DL research community itself may be viewed as a continually evolving, deep network of scientists who have influenced each other in complex ways. Starting from recent DL results, I tried to trace back the origins of relevant ideas through the past half century and beyond, sometimes using "local search" to follow citations of citations backwards in time. Since not all DL publications properly acknowledge earlier relevant work, additional global search strategies were employed, aided by consulting numerous neural network experts. As a result, the present preprint mostly consists of references. Nevertheless, through an expert selection bias I may have missed important work. A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century. For these reasons, this work should be viewed as merely a snapshot of an ongoing credit assignment process. To help improve it, please do not hesitate to send corrections and suggestions to juergen@idsia.ch.

show abstract

“…Because in real-life situations training data are always at a premium (Edelman, 2002), and because high-VCdim classifiers are too flexible and are therefore prone to overfitting (Baum and Haussler, 1989;Geman et al, 1992), it is necessary to break down the classification task into elements by relegating them to purposive visual subsystems. Being dedicated to a particular task (such as face recognition), such systems can afford to employ the simplest possible classifier that is up to the job.…”

Section: Representational Capacitymentioning

confidence: 99%