Exhaustive Learning

Schwartz, Daniel; Samalam, Vijay K.; Solla, Sara A.; Denker, John S.

doi:10.1162/neco.1990.2.3.374

Cited by 61 publications

(17 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Here too rational convergence appears to be a natural form, as suggested in general by [1], proved more rigorously by [12], and demonstrated in a specific case study by [19]. Given the much stronger assumptions of this model, it is not too surprising that many researchers have indicated a dichotomy between rational and exponential average case learning curves [3,23,24]. The common suggestion is that this dichotomy is determined by the existence of``gaps'' between target concepts.…”

Section: Significance and Related Workmentioning

confidence: 82%

“…In fact, it is not even obvious what property distinguishes rational from exponential convergence for simple chains in this case. It has often been suggested [3,23,24] that density in the metric d P should distinguish rational from exponential worst case learning curves. However, this suggestion can easily be shown to be false: Consider a concept chain (C, P) where C=[q=[0, q] : q # Q[0, 1]] consists of rational initial segments [0, q] of the unit interval X=[0, 1], and P is any distribution on [0, 1] such that P(q)>0 for all (and only) rational points q # Q[0, 1].…”

Section: Discussion and Research Directionsmentioning

confidence: 99%

See 1 more Smart Citation

Characterizing Rational versus Exponential Learning Curves

Schuurmans

1997

Journal of Computer and System Sciences

View full text Add to dashboard Cite

We consider the standard problem of learning a concept from random examples. Here a learning curve is defined to be the expected error of a learner's hypotheses as a function of training sample size. Haussler, Littlestone, and Warmuth have shown that, in the distribution-free setting, the smallest expected error a learner can achieve in the worst case over a class of concepts C converges rationally to zero error; i.e., 3(t &1 ) in the training sample size t. However, Cohn and Tesauro have recently demonstrated that exponential convergence can often be observed in experimental settings (i.e., average error decreasing as e 3( &t) ). By addressing a simple non-uniformity in the original analysis this paper shows how the dichotomy between rational and exponential worst case learning curves can be recovered in the distribution-free theory. In particular, our results support the experimental findings of Cohn and Tesauro: for finite concept classes any consistent learner achieves exponential convergence, even in the worst case, whereas for continuous concept classes no learner can exhibit sub-rational convergence for every target concept and domain distribution. We also draw a precise boundary between rational and exponential convergence for simple concept chains showing that somewheredense chains always force rational convergence in the worst case, while exponential convergence can always be achieved for nowheredense chains. ] 1997 Academic Press

show abstract

Section: Significance and Related Workmentioning

confidence: 82%

Section: Discussion and Research Directionsmentioning

confidence: 99%

Characterizing Rational versus Exponential Learning Curves

Schuurmans

1997

Journal of Computer and System Sciences

View full text Add to dashboard Cite

show abstract

“…Our formalism can be used to give a classification of the large-or asymptotics of scaled learning curves 7, thus completing a classification program that has been suggested by several researchers (Amari et al, 1992;Schwartz et al, 1990;Seung et al, 1992). From Eq.…”

Section: Large-or Asymptotics Of Scaled Learning Curvesmentioning

confidence: 92%

“…In this paper, we show that ideas from statistical mechanics (namely, the annealed approximation (Amari et al, 1992;Levin et al, 1989;Schwartz et al, 1990;Sompolinsky et al, 1991) and the thermodynamic limit (Sompolinsky et al, 1991)) can be used as the basis of a mathematically precise and rigorous theory of learning curves 3. This theory will be distribution-specific, but will not attempt to force a power law form on learning curves.…”

Section: Introductionmentioning

confidence: 99%

Rigorous learning curve bounds from statistical mechanics

Haussler

Kearns

Seung³

et al. 1997

Mach Learn

View full text Add to dashboard Cite

Abstract. In this paper we introduce and investigate a mathematically rigorous theory of learning curves that is based on ideas from statistical mechanics. The advantage of our theory over the well-established VapnikChervonenkis theory is that our bounds can be considerably tighter in many cases, and are also more reflective of the true behavior of learning curves. This behavior can often exhibit dramatic properties such as phase transitions, as well as power law asymptotics not explained by the VC theory. The disadvantages of our theory are that its application requires knowledge of the input distribution, and it is limited so far to finite cardinality function classes.We illustrate our results with many concrete examples of learning curve bounds derived from our theory.

show abstract

“…Figure 4 shows that this bound is indeed well above the actual learning curves if the number of weights is used as an approximate value of dvc. The learning curves are also well described by E c ( 1/(M + Mo), predicted by statistical learning theories for problems with continuous generalization spectrum (Schwartz et al 1990), and the algorithms are equally robust to limited training sets.…”

Section: Monte Car10 Studiesmentioning

confidence: 73%

Pattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior

Rögnvaldsson

1993

Neural Computation

View full text Add to dashboard Cite

The discrimination powers of multilayer perceptron (MLP) and learning vector quantization (LVQ) networks are compared for overlapping gaussian distributions. It is shown, both analytically and with Monte Carlo studies, that the MLP network handles high-dimensional problems in a more efficient way than LVQ. This is mainly due to the sigmoidal form of the MLP transfer function, but also to the fact that the MLP uses hyperplanes more efficiently. Both algorithms are equally robust to limited training sets and the learning curves fall off like UM, where M is the training set size, which is compared to theoretical predictions from statistical estimates and Vapnik-Chervonenkis bounds.

show abstract

Exhaustive Learning

Cited by 61 publications

References 5 publications

Characterizing Rational versus Exponential Learning Curves

Characterizing Rational versus Exponential Learning Curves

Rigorous learning curve bounds from statistical mechanics

Pattern Discrimination Using Feedforward Networks: A Benchmark Study of Scaling Behavior

Contact Info

Product

Resources

About