Dimension independent excess risk by stochastic gradient descent

Abstract

One classical canon of statistics is that large models are prone to overfitting, and model selection procedures are necessary for high dimensional data. However, many overparameterized models, such as neural networks, perform very well in practice, although they are often trained with simple online methods and regularization. The empirical success of overparameterized models, which is often known as benign overfitting, motivates us to have a new look at the statistical generalization theory for online optimization. In particular, we present a general theory on the excess risk of stochastic gradient descent (SGD) solutions for both convex and locally non-convex loss functions. We further discuss data and model conditions that lead to a "low effective dimension". Under these conditions, we show that the excess risk either does not depend on the ambient dimension p or depends on p via a poly-logarithmic factor. We also demonstrate that in several widely used statistical models, the "low effective dimension" arises naturally in overparameterized settings. The studied statistical applications include both convex models such as linear regression and logistic regression and non-convex models such as M -estimator and two-layer neural networks.

Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$

“…Developing an inferential theory for SGD becomes more challenging in particular in the growing-dimensional setting, when the number of parameters can grow with the number of iterations (or equivalently the number of observations used in online SGD). Such growing-dimensional settings are common in modern statistical machine learning problems and it well-known that online SGD has implicit regularization properties, as examined in several recent works including [1,68,74,70,19].…”

“…Beyond providing a theoretical framework for growing-dimensional inference, our results have practical implications for constructing algorithmic prediction intervals in linear regression. For a new test point a, independent of the training data used by SGD, choosing a in (2) directly yields a predictive confidence interval, complementing prior works on implicit regularization and benign overfitting [1,68,74,70,19]. Furthermore, our results can be used to develop algorithmic Wald-type tests for feature significance in highdimensional linear models-an essential tool in empirical sciences such as biology, social science, economics, and medicine [27,64,13].…”

Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$

“…Developing an inferential theory for SGD becomes more challenging in particular in the growing-dimensional setting, when the number of parameters can grow with the number of iterations (or equivalently the number of observations used in online SGD). Such growing-dimensional settings are common in modern statistical machine learning problems and it well-known that online SGD has implicit regularization properties, as examined in several recent works including [1,68,74,70,19].…”

“…Beyond providing a theoretical framework for growing-dimensional inference, our results have practical implications for constructing algorithmic prediction intervals in linear regression. For a new test point a, independent of the training data used by SGD, choosing a in (2) directly yields a predictive confidence interval, complementing prior works on implicit regularization and benign overfitting [1,68,74,70,19]. Furthermore, our results can be used to develop algorithmic Wald-type tests for feature significance in highdimensional linear models-an essential tool in empirical sciences such as biology, social science, economics, and medicine [27,64,13].…”

Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$

“…Developing an inferential theory for SGD becomes more challenging in particular in the growing-dimensional setting, when the number of parameters can grow with the number of iterations (or equivalently the number of observations used in online SGD). Such growing-dimensional settings are common in modern statistical machine learning problems and it well-known that online SGD has implicit regularization properties, as examined in several recent works including [1,68,74,70,19].…”

“…Beyond providing a theoretical framework for growing-dimensional inference, our results have practical implications for constructing algorithmic prediction intervals in linear regression. For a new test point a, independent of the training data used by SGD, choosing a in (2) directly yields a predictive confidence interval, complementing prior works on implicit regularization and benign overfitting [1,68,74,70,19]. Furthermore, our results can be used to develop algorithmic Wald-type tests for feature significance in highdimensional linear models-an essential tool in empirical sciences such as biology, social science, economics, and medicine [27,64,13].…”

Statistical Inference for Linear Functionals of Online Least-squares SGD when $t \gtrsim d^{1+δ}$

“…Developing an inferential theory for SGD becomes more challenging in particular in the growing-dimensional setting, when the number of parameters can grow with the number of iterations (or equivalently the number of observations used in online SGD). Such growing-dimensional settings are common in modern statistical machine learning problems and it well-known that online SGD has implicit regularization properties, as examined in several recent works including [1,68,74,70,19].…”

“…Beyond providing a theoretical framework for growing-dimensional inference, our results have practical implications for constructing algorithmic prediction intervals in linear regression. For a new test point a, independent of the training data used by SGD, choosing a in (2) directly yields a predictive confidence interval, complementing prior works on implicit regularization and benign overfitting [1,68,74,70,19]. Furthermore, our results can be used to develop algorithmic Wald-type tests for feature significance in highdimensional linear models-an essential tool in empirical sciences such as biology, social science, economics, and medicine [27,64,13].…”