Tight Nonparametric Convergence Rates for Stochastic Gradient Descent under the Noiseless Linear Model

Abstract

In the context of statistical supervised learning, the noiseless linear model assumes that there exists a deterministic linear relation Y = θ * , Φ(U ) between the random output Y and the random feature vector Φ(U ), a potentially non-linear transformation of the inputs U . We analyze the convergence of single-pass, fixed step-size stochastic gradient descent on the least-square risk under this model. The convergence of the iterates to the optimum θ * and the decay of the generalization error follow polynomial convergence rates with exponents that both depend on the regularities of the optimum θ * and of the feature vectors Φ(U ). We interpret our result in the reproducing kernel Hilbert space framework; as a special case, we analyze an online algorithm for estimating a real function on the unit interval from the noiseless observation of its value at randomly sampled points. The convergence depends on the Sobolev smoothness of the function and of a chosen kernel. Finally, we apply our analysis beyond the supervised learning setting to obtain convergence rates for the averaging process (a.k.a. gossip algorithm) on a graph depending on its spectral dimension.

Benign Overfitting of Constant-Stepsize SGD for Linear Regression

“…Our variance analysis is sharper than Dieuleveut and Bach (2015) in that we provide a bound in terms of the full spectrum (with the same R 2 assumption) along with a lower bound, while Dieuleveut and Bach (2015) assume decay conditions on the spectrum. The bias analysis in Dieuleveut and Bach (2015); Berthier et al (2020) relies on a stronger assumption in that H −α w * 2 must be finite, where α > 0 is a constant (e.g., A4 in Bach and Moulines (2013) and Theorem 1 condition (a) in Berthier et al (2020)); our conjecture is that without relying on stronger fourth moment assumption (such as those consistent with sub-Gaussians), such dependencies are not avoidable. Our fourth moment assumption is a natural starting point for analyzing the over-parameterized regime because it also allows for direct comparisons to OLS and ridge regression, as discussed above.…”

“…We make this assumption to emphasize that our variance analysis does not rely on stronger assumptions than those in a number of prior works for iterate averaged SGD (Bach and Moulines, 2013;Jain et al, 2017b;Berthier et al, 2020). Moreover, note that this assumption is implied by Assumption 2.2 by setting A = I, which gives R 2 = α tr(H).…”

“…We further develop on the proof techniques in Jain et al (2017a), where we use properties of asymptotic stationary distributions for the purposes of finite sample size analysis. We now discuss related works in the overparameterized regime (Dieuleveut and Bach, 2015;Berthier et al, 2020) R 2 H, as adopted in Bach and Moulines (2013); Défossez and Bach (2015); Dieuleveut et al (2017); Jain et al (2017a,b). Our assumption implies an R 2 bound with R 2 = α tr(H).…”

“…For the case of finite dimensional, linear regression, there are a number of more modern proofs which provide finite, non-asymptotic rates (Défossez and Bach, 2015;Bach and Moulines, 2013;Dieuleveut et al, 2017;Jain et al, 2017b,a). With regards to the overparameterized regime, there is far less work (Dieuleveut and Bach, 2015;Berthier et al, 2020) being notable exceptions. (See Section 3 for further discussion on these related works.…”

Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions

“…More refined assumptions can lead to completely different convergence rates. In the context of kernel methods and SGD studies, it is common to consider power law spectral conditions ("source condition" and "capacity condition") that are known to give power law convergence rate bounds O(n −ξ ) with different exponents ξ (Berthier et al, 2020;Zou et al, 2021;Varre et al, 2021).…”