The Hilbert Kernel Regression Estimate

Devroye, Luc; Györfi, László; Krzyżak, Adam

doi:10.1006/jmva.1997.1725

Cited by 27 publications

(69 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, as noted by Devroye, Györfi and Krzyżak (1998), a kernel that is singular at 0 does interpolate the data. While the Hilbert kernel K(u) = u −d 2 , suggested in Devroye et al (1998), does not enjoy non-asymptotic rates of convergence, its truncated version…”

Section: Local Methods: Nadaraya-watsonmentioning

confidence: 99%

Deep learning: a statistical viewpoint

2021

View full text Add to dashboard Cite

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behaviour of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favourable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

show abstract

Section: Local Methods: Nadaraya-watsonmentioning

confidence: 99%

Deep learning: a statistical viewpoint

2021

View full text Add to dashboard Cite

show abstract

“…Traditionally, consistency and rates of convergence have been a central object of statistical investigation. The first result in this direction is Devroye et al (1998), which showed statistical consistency of a certain kernel regression scheme, closely related to Shepard's inverse distance interpolation (Shepard 1968).…”

Section: Optimality Of K-nn With Singular Weighting Schemesmentioning

confidence: 97%

“…It is perhaps ironic that an outlier feature of the 1-NN rule, shared with no other common methods in the classical statistics literature (except for the relatively unknown work by Devroye, Györfi and Krzyżak 1998), may be one of the cues to understanding modern deep learning.…”

Section: The Peculiar Case Of 1-nnmentioning

confidence: 99%

Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation

Belkin¹

2021

Acta Numerica

View full text Add to dashboard Cite

In the past decade the mathematical theory of machine learning has lagged far behind the triumphs of deep neural networks on practical challenges. However, the gap between theory and practice is gradually starting to close. In this paper I will attempt to assemble some pieces of the remarkable and still incomplete mathematical mosaic emerging from the efforts to understand the foundations of deep learning. The two key themes will be interpolation and its sibling over-parametrization. Interpolation corresponds to fitting data, even noisy data, exactly. Over-parametrization enables interpolation and provides flexibility to select a suitable interpolating model.As we will see, just as a physical prism separates colours mixed within a ray of light, the figurative prism of interpolation helps to disentangle generalization and optimization properties within the complex picture of modern machine learning. This article is written in the belief and hope that clearer understanding of these issues will bring us a step closer towards a general theory of deep learning and machine learning.

show abstract

“…Rather than to prove the full-blown universal theorem, we restrict ourselves to the uniform density on the real line and recall the following result from Devroye and Krzyżak (1998), which is applicable as for d=1, the Hilbert product kernel estimate coincides with the standard Hilbert kernel estimate. …”

Section: Lack Of Strong Convergencementioning

confidence: 97%