Rigorous learning curve bounds from statistical mechanics

Haussler, David; Kearns, Michael; Seung, H. Sebastian; Tishby, Naftali

doi:10.1007/bf00114010

Cited by 38 publications

(72 citation statements)

References 30 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Bias does not vary with training data size n, but the error due to variance should decrease as O( 1 / √ n) if the training observations are independent (Domingos, 2000a,b). The power-law models used in this paper have been investigated many times in prior literature (Haussler et al, 1996;Mukherjee et al, 2003;Figueroa et al, 2012;Beleites et al, 2013;Hajian-Tilaki, 2014;Cho et al, 2015). Sun et al (2017), Barone et al (2017) and the concurrent unpublished work by Hestness et al (2017) point out that these power-law models describe modern ML and NLP systems quite well, including complex deep-learning systems, so we expect our results to generalise to these systems.…”

Section: Related Workmentioning

confidence: 99%

Predicting accuracy on large datasets from smaller pilot data

Johnson¹,

Anderson

Dras³

et al. 2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

Because obtaining training data is often the most difficult part of an NLP or ML project, we develop methods for predicting how much data is required to achieve a desired test accuracy by extrapolating results from systems trained on a small pilot training dataset. We model how accuracy varies as a function of training size on subsets of the pilot data, and use that model to predict how much training data would be required to achieve the desired accuracy. We introduce a new performance extrapolation task to evaluate how well different extrapolations predict system accuracy on larger training sets. We show that details of hyperparameter optimisation and the extrapolation models can have dramatic effects in a document classification task. We believe this is an important first step in developing methods for estimating the resources required to meet specific engineering performance targets.

show abstract

Section: Related Workmentioning

confidence: 99%

Predicting accuracy on large datasets from smaller pilot data

Johnson¹,

Anderson

Dras³

et al. 2018

Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

show abstract

“…On the other hand, knowing that we would like a particular quality guarantee, we can ask how large a sample we need to draw to ensure that guarantee. The former question has been addressed for predictive learning in work on self-bounding learning algorithms [4] and shell decomposition bounds [7,11].…”

Section: Prior Workmentioning

confidence: 99%

A Scalable Constant-Memory Sampling Algorithm for Pattern Discovery in Large Databases

Scheffer

Wrobel

2002

Principles of Data Mining and Knowledge Discovery

View full text Add to dashboard Cite

Many data mining tasks can be seen as an instance of the problem of finding the most interesting (according to some utility function) patterns in a large database. In recent years, significant progress has been achieved in scaling algorithms for this task to very large databases through the use of sequential sampling techniques. However, except for sampling-based greedy algorithms which cannot give absolute quality guarantees, the scalability of existing approaches to this problem is only with respect to the data, not with respect to the size of the pattern space: it is universally assumed that the entire hypothesis space fits in main memory. In this paper, we describe how this class of algorithms can be extended to hypothesis spaces that do not fit in memory while maintaining the algorithms' precise ε − δ quality guarantees. We present a constant memory algorithm for this task and prove that it possesses the required properties. In an empirical comparison, we compare variable memory and constant memory sampling.

show abstract

“…This has been highlighted perhaps most prominently in recent work on neural network models, in which the model complexity and data size increase together. For this reason, the double asymptotic regime where n, N → ∞, with N/n → c, a constant, is a particularly interesting (and likely more realistic) limit, despite being technically more challenging [48,51,21,15,37,32,5]. In particular, working in this regime allows for a finer quantitative assessment of machine learning systems, as a function of their relative complexity N/n, as well as for a precise description of the under-to over-parameterized "phase transition" (that does not appear in the N → ∞ alone analysis).…”

Section: Introductionmentioning

confidence: 99%

A random matrix analysis of random Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent*

2021

View full text Add to dashboard Cite

This article characterizes the exact asymptotics of random Fourier feature (RFF) regression, in the realistic setting where the number of data samples n, their dimension p, and the dimension of feature space N are all large and comparable. In this regime, the random RFF Gram matrix no longer converges to the well-known limiting Gaussian kernel matrix (as it does when N → ∞ alone), but it still has a tractable behavior that is captured by our analysis. This analysis also provides accurate estimates of training and test regression errors for large n, p, N. Based on these estimates, a precise characterization of two qualitatively different phases of learning, including the phase transition between them, is provided; and the corresponding double descent test error curve is derived from this phase transition behavior. These results do not depend on strong assumptions on the data distribution, and they perfectly match empirical results on real-world data sets.

show abstract

Rigorous learning curve bounds from statistical mechanics

Cited by 38 publications

References 30 publications

Predicting accuracy on large datasets from smaller pilot data

Predicting accuracy on large datasets from smaller pilot data

A Scalable Constant-Memory Sampling Algorithm for Pattern Discovery in Large Databases

A random matrix analysis of random Fourier features: beyond the Gaussian kernel, a precise phase transition, and the corresponding double descent*

Contact Info

Product

Resources

About