Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

Abstract

An evolving line of machine learning works observe empirical evidence that suggests interpolating estimators -the ones that achieve zero training error -may not necessarily be harmful. This paper pursues theoretical understanding for an important type of interpolators: the minimum 1-norm interpolator, which is motivated by the observation that several learning algorithms favor low 1-norm solutions in the over-parameterized regime. Concretely, we consider the noisy sparse regression model under Gaussian design, focusing on linear sparsity and high-dimensional asymptotics (so that both the number of features and the sparsity level scale proportionally with the sample size).We observe, and provide rigorous theoretical justification for, a curious multi-descent phenomenon; that is, the generalization risk of the minimum 1-norm interpolator undergoes multiple (and possibly more than two) phases of descent and ascent as one increases the model capacity. This phenomenon stems from the special structure of the minimum 1-norm interpolator as well as the delicate interplay between the over-parameterized ratio and the sparsity, thus unveiling a fundamental distinction in geometry from the minimum 2-norm interpolator. Our finding is built upon an exact characterization of the risk behavior, which is governed by a system of two non-linear equations with two unknowns.

Mitigating multiple descents: A model-agnostic framework for risk monotonization

“…Figure 1 illustrates the above result for the minimum 2 -norm least squares estimator (Hastie et al, 2019) and the minimum 1 -norm least squares estimator (Li and Wei, 2021). The light-blue lines show the asymptotic risk profiles of the two procedures, which are non-monotonic as they diverge to infinity around the interpolation threshold of 1, at which the sample size and the number of features are equal.…”

“…The MN1LS estimator connects naturally to the basis pursuit estimator in compressed sensing literature (e.g. Candes and Tao (2006); Donoho (2006)) and its risk in the proportional regime has been recently analyzed in Mitra (2019); Li and Wei (2021). The MN1LS predictor is now defined as…”

“…Past the interpolation threshold, the test error tapers down as the complexity of the algorithm increases relative to the sample size. Furthermore, for some algorithms and settings, e.g., the lasso and the minimum 1 -norm least square (e.g., Li and Wei, 2021) or various structures of the design matrix (Adlam and Pennington, 2020;Chen et al, 2020), multiple descents may occur. Double and multiple descent phenomena have been first demonstrated empirically, e.g., for decision trees, random features and two-layer and deep neural networks, and some of these findings have now been corroborated by rigorous theories in a growing body of work: see, e.g., Neyshabur et al (2014); Nakkiran et al (2019); Belkin et al (2018bBelkin et al ( , 2019a; Mei and Montanari (2019); Adlam and Pennington (2020); Chen et al (2020); Li and Wei (2021), among others.…”

“…Furthermore, for some algorithms and settings, e.g., the lasso and the minimum 1 -norm least square (e.g., Li and Wei, 2021) or various structures of the design matrix (Adlam and Pennington, 2020;Chen et al, 2020), multiple descents may occur. Double and multiple descent phenomena have been first demonstrated empirically, e.g., for decision trees, random features and two-layer and deep neural networks, and some of these findings have now been corroborated by rigorous theories in a growing body of work: see, e.g., Neyshabur et al (2014); Nakkiran et al (2019); Belkin et al (2018bBelkin et al ( , 2019a; Mei and Montanari (2019); Adlam and Pennington (2020); Chen et al (2020); Li and Wei (2021), among others. However, in general, the shape and number of local minima associated with a non-monotonic risk profile due to double descent depend non-trivially on the learning problem, the algorithm deployed, and to an extent, the properties of the data generating distribution in ways that are only partially understood.…”

A New Central Limit Theorem for the Augmented IPW Estimator: Variance Inflation, Cross-Fit Covariance and Beyond

“…We analyze the cross-fitted AIPW estimator in the "proportional asymptotic regime", where the number of observations n and features p both diverge, with the ratio p/n converging to some constant κ > 0. This regime has attracted considerable recent attention in high-dimensional statistics [12, 13, 15-18, 23, 25, 29, 30, 38-40, 42, 44, 46, 55, 59, 61, 65, 82, 84, 94, 102, 104, 107, 109, 115, 117, 118], statistical machine learning and analysis of algorithms [31,36,60,67,68,71,72,74,79], econometrics [4,5,14,[26][27][28]51] etc, and shares roots with probability theory and statistical physics [78,120]. Asymptotic approximations derived under this regime demonstrate commendable performance even under moderate sample sizes (c.f.…”