On the robustness of minimum norm interpolators and regularized empirical risk minimizers

Chinot, Geoffrey; Löffler, Matthias; Geer, Sara van de

doi:10.1214/22-aos2190

Cited by 5 publications

(5 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that we used the majorizing measure theorem [42] to get the equivalence between Talagrand's γ 2 -functional and the Gaussian mean width and that for the Gaussian measure µ, the Orlicz space L ψ 2 (µ) is equivalent to the Hilbert space L 2 (µ). One can check that diam(F, L 2 (µ)) = 1 and the result follows for t = N/(64C 2 ), κ RIP = 1/[8 √ C] and the definition of R N (Γ 1/2 B) in (14).…”

Section: Isomorphy and Restricted Isomorphy Propertiesmentioning

confidence: 88%

“…Theorem 6. There are absolute constants 0 < κ RIP < 1 and c 0 such that for the fixed point R N (Γ 1/2 B) defined in (14) and X 1 = G (N ×p) Γ 1/2 , the following holds. With probability at least…”

Section: Isomorphy and Restricted Isomorphy Propertiesmentioning

confidence: 99%

“…2 ) = 0 which means that X 1 satisfies an isomorphic property (see (15)) over the entire space range(Γ). We will use this property to study β in the case where k ≤ κ RIP N ; we therefore state it in the following result.…”

Section: Isomorphy and Restricted Isomorphy Propertiesmentioning

confidence: 99%

“…when an estimator with a perfect fit on the training data (called interpolant estimator) can still generalize well. Several results have been obtained on BO in various statistical frameworks such as regression [8,9,38,7,18,37,49,14,52,29,32,15,53,11,38,22] and classification [8,41,12,30,48]. The whole point is to find conditions under which interpolant estimators can generalize well (that is to predict well on out-of-sample data).…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A geometrical viewpoint on the benign overfitting property of the minimum ℓ 2 -norm interpolant estimator and its universality

Lecué,

Shang

2023

Preprint

View full text Add to dashboard Cite

In the linear regression model, the minimum ℓ 2-norm interpolant estimatorˆβestimatorˆ estimatorˆβ has received much attention since it was proved to be consistent even though it fits noisy data perfectly under some condition on the covariance matrix Σ of the input vector, known as benign overfitting. Motivated by this phenomenon, we study the generalization property of this estimator from a geometrical viewpoint. Our main results extend and improve the convergence rates as well as the deviation probability from [44]. Our proof differs from the classical bias/variance analysis and is based on the self-induced regularization property introduced in [3]: ˆ β can be written as a sum of a ridge estimatorˆβestimatorˆ estimatorˆβ 1:k and an overfitting componentˆβcomponentˆ componentˆβ k+1:p which follows a decomposition of the features space R p = V 1:k ⊕ ⊥ V k+1:p into the space V 1:k spanned by the top k eigenvectors of Σ and V k+1:p spanned by the p − k last ones. We also prove a matching lower bound for the expected prediction risk thus obtain the sufficient and necessary conditions for benign overfitting ofˆβofˆ ofˆβ. The two geometrical properties of random Gaussian matrices at the heart of our analysis are the Dvoretsky-Milman theorem and isomorphic and restricted isomorphic properties. In particular, the Dvoretsky dimension appearing naturally in our geometrical viewpoint, coincides with the effective rank from [2, 44] and is the key tool for handling the behavior of the design matrix restricted to the sub-space V k+1:p where overfitting happens. We extend these results to heavy-tailed scenarii proving the universality of this phenomenon beyond exponential moment assumptions. This phenomenon is unknown before and is widely believed to be a significant challenge. This follows from an anistropic version of the probabilistic Dvoretsky-Milman theorem that holds for heavy-tailed vectors which is of independent interest.

show abstract

Section: Isomorphy and Restricted Isomorphy Propertiesmentioning

confidence: 88%

Section: Isomorphy and Restricted Isomorphy Propertiesmentioning

confidence: 99%

Section: Isomorphy and Restricted Isomorphy Propertiesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A geometrical viewpoint on the benign overfitting property of the minimum ℓ 2 -norm interpolant estimator and its universality

Lecué,

Shang

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Because of its simplicity and well-develped theory in classical machine learning [58,15,16], sparse modeling is often used to provide theoretical understanding of modern large and over-parameterized models. This include works on implicit regularization [59,60,61,62,63], nonconvex optimization [64,65], noise interpolators [66,67,68], etc. However, the aforementioned work uses sparsity as a testbed or toy model to gain insights, without implication of existence of sparsity in DNNs.…”

Section: Sparsity For Robustnessmentioning

confidence: 99%

Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers

Li¹,

You²,

Bhojanapalli³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper studies the curious phenomenon for machine learning models with Transformer architectures that their activation maps are sparse. By activation map we refer to the intermediate output of the multi-layer perceptrons (MLPs) after a ReLU activation function, and by "sparse" we mean that on average very few entries (e.g., 3.0% for T5-Base and 6.3% for ViT-B16) are nonzero for each input to MLP. Moreover, larger Transformers with more layers and wider MLP hidden dimensions are sparser as measured by the percentage of nonzero entries. Through extensive experiments we demonstrate that the emergence of sparsity is a prevalent phenomenon that occurs for both natural language processing and vision tasks, on both training and evaluation data, for Transformers of various configurations, at layers of all depth levels, as well as for other architectures including MLP-mixers and 2-layer MLPs. We show that sparsity also emerges using training datasets with random labels, or with random inputs, or with infinite amount of data, demonstrating that sparsity is not a result of a specific family of datasets. We discuss how sparsity immediately implies a way to significantly reduce the FLOP count and improve efficiency for Transformers. Moreover, we demonstrate perhaps surprisingly that enforcing an even sparser activation via Top-k thresholding with a small value of k brings a collection of desired but missing properties for Transformers, namely less sensitivity to noisy training data, more robustness to input corruptions, and better calibration for their prediction confidence.

show abstract

Tractability from overparametrization: the example of the negative perceptron

Montanari,

Zhong,

Zhou

2024

Probab. Theory Relat. Fields

View full text Add to dashboard Cite

On the robustness of minimum norm interpolators and regularized empirical risk minimizers

Cited by 5 publications

References 39 publications

A geometrical viewpoint on the benign overfitting property of the minimum ℓ 2 -norm interpolant estimator and its universality

A geometrical viewpoint on the benign overfitting property of the minimum ℓ 2 -norm interpolant estimator and its universality

Large Models are Parsimonious Learners: Activation Sparsity in Trained Transformers

Tractability from overparametrization: the example of the negative perceptron

Contact Info

Product

Resources

About