Precise Error Analysis of Regularized M-estimators in High-dimensions

Thrampoulidis, Christos; Abbasi, Ehsan; Hassibi, Babak

doi:10.48550/arxiv.1601.06233

Cited by 7 publications

(34 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our predictions show that the asymptotic limit of the training and generalization errors can be precisely predicted after solving a scalar deterministic formulation. The theoretical predictions are obtained using an extended version of the convex Gaussian min-max theorem (CGMT) [10], [11] which we refer to as the multivariate CGMT. The new version of the CGMT accounts for the correlation introduced by injecting Gaussian noise during the learning process.…”

Section: Contributionsmentioning

confidence: 99%

“…Our analysis is based on an extended version of the CGMT referred to as the multivariate CGMT. The CGMT is first used in [20] and further developed in [10]. It extends a Gaussian theorem first introduced in [21].…”

Section: Related Workmentioning

confidence: 99%

“…It relies on (strong) convexity properties to prove an equivalence between two Gaussian processes. It has been successfully applied in the analysis of convex regression [10], [16], [22] and convex classification [23]- [26] formulations.…”

Section: Related Workmentioning

confidence: 99%

“…, where y = ϕ(s), and s is a standard Gaussian random variable. Note that the problem defined in (10) depends on q ⋆ t,τ which is given by…”

Section: A Precise Asymptotic Analysismentioning

confidence: 99%

“…where C ⋆ (∆, λ) is the optimal cost of the deterministic problem in (10). Here, the function h(•) is defined as follows…”

Section: A Precise Asymptotic Analysismentioning

confidence: 99%

See 4 more Smart Citations

On the Inherent Regularization Effects of Noise Injection During Training

Dhifallah,

2021

Preprint

View full text Add to dashboard Cite

Randomly perturbing networks during the training process is a commonly used approach to improving generalization performance. In this paper, we present a theoretical study of one particular way of random perturbation, which corresponds to injecting artificial noise to the training data. We provide a precise asymptotic characterization of the training and generalization errors of such randomly perturbed learning problems on a random feature model. Our analysis shows that Gaussian noise injection in the training process is equivalent to introducing a weighted ridge regularization, when the number of noise injections tends to infinity. The explicit form of the regularization is also given. Numerical results corroborate our asymptotic predictions, showing that they are accurate even in moderate problem dimensions. Our theoretical predictions are based on a new correlated Gaussian equivalence conjecture that generalizes recent results in the study of random feature models.

show abstract

Section: Contributionsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

“…, where y = ϕ(s), and s is a standard Gaussian random variable. Note that the problem defined in (10) depends on q ⋆ t,τ which is given by…”

Section: A Precise Asymptotic Analysismentioning

confidence: 99%

“…where C ⋆ (∆, λ) is the optimal cost of the deterministic problem in (10). Here, the function h(•) is defined as follows…”

Section: A Precise Asymptotic Analysismentioning

confidence: 99%

See 3 more Smart Citations

On the Inherent Regularization Effects of Noise Injection During Training

Dhifallah,

2021

Preprint

View full text Add to dashboard Cite

show abstract

Replica analysis of overfitting in regression models for time to event data: the impact of censoring

Massa,

Mozeika,

Coolen

2024

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

We use statistical mechanics techniques, viz. the replica method, to model the effect of censoring on overfitting in Cox's proportional hazards model, the dominant regression method for time-to-event data. In the overfitting regime, Maximum Likelihood parameter estimators are known to be biased already for small values of the ratio of the number of covariates over the number of samples. The inclusion of censoring was avoided in previous overfitting analyses for mathematical convenience, but is vital to make any theory applicable to real-world medical data, where censoring is ubiquitous. Upon constructing efficient algorithms for solving the new (and more complex) RS equations and comparing the solutions with numerical simulation data, we find excellent agreement, even for large censoring rates. We then address the practical problem of using the theory to correct the biased ML estimators {without} knowledge of the data-generating distribution. This is achieved via a novel numerical algorithm that self-consistently approximates all relevant parameters of the data generating distribution while simultaneously solving the RS equations. We investigate numerically the statistics of the corrected estimators, and show that the proposed new algorithm indeed succeeds in removing the bias of the ML estimators, for both the association parameters and for the cumulative hazard.

show abstract

Low noise sensitivity analysis of -minimization in oversampled systems

Weng

Maleki

2019

Information and Inference: A Journal of the IMA

View full text Add to dashboard Cite

The class of ℓ q -regularized least squares (LQLS) are considered for estimating β ∈ R p from its n noisy linear observations y = Xβ + w. The performance of these schemes are studied under the high-dimensional asymptotic setting in which the dimension of the signal grows linearly with the number of measurements. In this asymptotic setting, phase transition diagrams (PT) are often used for comparing the performance of different estimators. PT specifies the minimum number of observations required by a certain estimator to recover a structured signal, e.g. a sparse one, from its noiseless linear observations. Although phase transition analysis is shown to provide useful information for compressed sensing, the fact that it ignores the measurement noise not only limits its applicability in many application areas, but also may lead to misunderstandings. For instance, consider a linear regression problem in which n > p and the signal is not exactly sparse. If the measurement noise is ignored in such systems, regularization techniques, such as LQLS, seem to be irrelevant since even the ordinary least squares (OLS) returns the exact solution. However, it is well-known that if n is not much larger than p then the regularization techniques improve the performance of OLS.In response to this limitation of PT analysis, we consider the low-noise sensitivity analysis. We show that this analysis framework (i) reveals the advantage of LQLS over OLS, (ii) captures the difference between different LQLS estimators even when n > p, and (iii) provides a fair comparison among different estimators in high signal-to-noise ratios. As an application of this framework, we will show that under mild conditions LASSO outperforms other LQLS even when the signal is dense. Finally, by a simple transformation we connect our low-noise sensitivity framework to the classical asymptotic regime in which n/p → ∞ and characterize how and when regularization techniques offer improvements over ordinary least squares, and which regularizer gives the most improvement when the sample size is large.1. It reveals certain phenomena that are important in applications and are not captured by PT analysis. For instance, one immediately sees the impact of the regularizer and the magnitudes of the elements of β on the AMSE. Furthermore, these relations are expressed explicitly and can be interpreted easily.2. It provides a bridge between the phase transition analysis proposed in compressed sensing, and the classical large sample-size asymptotic (n/p → ∞). We will discuss some of the implications of this connection for the classical asymptotics in Section 3.3.As a consequence, low noise sensitivity analysis enables us to present a fair comparison among different LQLS, and reveal different factors that affect their performance.

show abstract

Precise Error Analysis of Regularized M-estimators in High-dimensions

Cited by 7 publications

References 0 publications

On the Inherent Regularization Effects of Noise Injection During Training

On the Inherent Regularization Effects of Noise Injection During Training

Replica analysis of overfitting in regression models for time to event data: the impact of censoring

Low noise sensitivity analysis of -minimization in oversampled systems

Contact Info

Product

Resources

About