Robustness in sparse high-dimensional linear models: Relative efficiency and robust approximate message passing

Bradić, Jelena

doi:10.1214/16-ejs1212

Cited by 16 publications

(32 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…13 play a crucial role in our analysis; the connections are detailed in SI Appendix . While this paper was under review, we also learned about extensions to penalized versions of such strongly convex losses (15). Again, this literature is concerned with linear models only and it is natural to wonder what extensions to generalized linear models might look like; see the comments at the end of the talk (16).…”

Section: Prior Workmentioning

confidence: 99%

A modern maximum-likelihood theory for high-dimensional logistic regression

Sur

Candès

2019

Proc. Natl. Acad. Sci. U.S.A.

187

185

View full text Add to dashboard Cite

Students in statistics or data science usually learn early on that when the sample size n is large relative to the number of variables p, fitting a logistic model by the method of maximum likelihood produces estimates that are consistent and that there are well-known formulas that quantify the variability of these estimates which are used for the purpose of statistical inference. We are often told that these calculations are approximately valid if we have 5 to 10 observations per unknown parameter. This paper shows that this is far from the case, and consequently, inferences produced by common software packages are often unreliable. Consider a logistic model with independent features in which n and p become increasingly large in a fixed ratio. We prove that (i) the maximum-likelihood estimate (MLE) is biased, (ii) the variability of the MLE is far greater than classically estimated, and (iii) the likelihood-ratio test (LRT) is not distributed as a χ2. The bias of the MLE yields wrong predictions for the probability of a case based on observed values of the covariates. We present a theory, which provides explicit expressions for the asymptotic bias and variance of the MLE and the asymptotic distribution of the LRT. We empirically demonstrate that these results are accurate in finite samples. Our results depend only on a single measure of signal strength, which leads to concrete proposals for obtaining accurate inference in finite samples through the estimate of this measure.

show abstract

Section: Prior Workmentioning

confidence: 99%

A modern maximum-likelihood theory for high-dimensional logistic regression

Sur

Candès

2019

Proc. Natl. Acad. Sci. U.S.A.

187

185

View full text Add to dashboard Cite

show abstract

“…Linear regression with random design G in the high-dimensional regime M, N → +∞ proportionally has attracted a lot of attention, mainly in Bayes-optimal settings [6,4,5,25], or in the context of robust statistics and M-estimation, where an estimator is obtained as the mode of a log-concave posterior measure (or equivalently as the minimizer of a convex cost function) [31,8,13,2,1,9,29,11,12,17,18,3,34,33,30]. A main difference between the literature on M-estimation and the present contribution is that we analyse a "finite temperature" estimator, namely, the mean of a log-concave posterior measure instead of its mode.…”

Section: Introductionmentioning

confidence: 99%

Performance of Bayesian linear regression in a model with mismatch

Barbier¹,

Chen²,

Panchenko³

et al. 2021

Preprint

View full text Add to dashboard Cite

For a model of high-dimensional linear regression with random design, we analyze the performance of an estimator given by the mean of a log-concave Bayesian posterior distribution with gaussian prior. The model is mismatched in the following sense: like the model assumed by the statistician, the labels-generating process is linear in the input data, but both the classifier ground-truth prior and gaussian noise variance are unknown to her. This inference model can be rephrased as a version of the Gardner model in spin glasses and, using the cavity method, we provide fixed point equations for various overlap order parameters, yielding in particular an expression for the mean-square reconstruction error on the classifier (under an assumption of uniqueness of solutions). As a direct corollary we obtain an expression for the free energy. Similar models have already been studied by Shcherbina and Tirozzi and by Talagrand, but our arguments are more straightforward and some assumptions are relaxed. An interesting consequence of our analysis is that in the random design setting of ridge regression, the performance of the posterior mean is independent of the noise variance (or "temperature") assumed by the statistician, and matches the one of the usual (zero temperature) ridge estimator.

show abstract

“…Donoho and Tanner characterized the phase transition curve for LASSO and some of its variants. Inspired by Donoho and Tanner's breakthrough, many researchers explored the performance of different algorithms under the asymptotic settings n/p → δ and p → ∞ Reeves and Donoho (2013); Stojnic (2009bStojnic ( ,a, 2013; Amelunxen et al (2014); Thrampoulidis et al (2016); El Karoui et al (2013); Karoui (2013); ; ; Donoho and Montanari (2015); Bradic and Chen (2015); Donoho et al (2011a,b); Foygel and Mackey (2014); Zheng et al (2017); Rangan et al (2009); Krzakala et al (2012); Bayati and Montanri (2011); Bayati and Montanari (2012); Reeves and Gastpar (2008); Reeves and Pfister (2016). In this paper, we use the message passing analysis that was developed in a series of papers Donoho et al (2011bDonoho et al ( , 2009); Maleki (2010); Bayati and Montanri (2011); Bayati and Montanari (2012); Maleki et al (2013) to characterize the asymptotic mean square errors of LQLS.…”

Section: Related Workmentioning

confidence: 99%

Low noise sensitivity analysis of -minimization in oversampled systems

Weng

Maleki

2019

Information and Inference: A Journal of the IMA

View full text Add to dashboard Cite

The class of ℓ q -regularized least squares (LQLS) are considered for estimating β ∈ R p from its n noisy linear observations y = Xβ + w. The performance of these schemes are studied under the high-dimensional asymptotic setting in which the dimension of the signal grows linearly with the number of measurements. In this asymptotic setting, phase transition diagrams (PT) are often used for comparing the performance of different estimators. PT specifies the minimum number of observations required by a certain estimator to recover a structured signal, e.g. a sparse one, from its noiseless linear observations. Although phase transition analysis is shown to provide useful information for compressed sensing, the fact that it ignores the measurement noise not only limits its applicability in many application areas, but also may lead to misunderstandings. For instance, consider a linear regression problem in which n > p and the signal is not exactly sparse. If the measurement noise is ignored in such systems, regularization techniques, such as LQLS, seem to be irrelevant since even the ordinary least squares (OLS) returns the exact solution. However, it is well-known that if n is not much larger than p then the regularization techniques improve the performance of OLS.In response to this limitation of PT analysis, we consider the low-noise sensitivity analysis. We show that this analysis framework (i) reveals the advantage of LQLS over OLS, (ii) captures the difference between different LQLS estimators even when n > p, and (iii) provides a fair comparison among different estimators in high signal-to-noise ratios. As an application of this framework, we will show that under mild conditions LASSO outperforms other LQLS even when the signal is dense. Finally, by a simple transformation we connect our low-noise sensitivity framework to the classical asymptotic regime in which n/p → ∞ and characterize how and when regularization techniques offer improvements over ordinary least squares, and which regularizer gives the most improvement when the sample size is large.1. It reveals certain phenomena that are important in applications and are not captured by PT analysis. For instance, one immediately sees the impact of the regularizer and the magnitudes of the elements of β on the AMSE. Furthermore, these relations are expressed explicitly and can be interpreted easily.2. It provides a bridge between the phase transition analysis proposed in compressed sensing, and the classical large sample-size asymptotic (n/p → ∞). We will discuss some of the implications of this connection for the classical asymptotics in Section 3.3.As a consequence, low noise sensitivity analysis enables us to present a fair comparison among different LQLS, and reveal different factors that affect their performance.

show abstract

Robustness in sparse high-dimensional linear models: Relative efficiency and robust approximate message passing

Cited by 16 publications

References 41 publications

A modern maximum-likelihood theory for high-dimensional logistic regression

A modern maximum-likelihood theory for high-dimensional logistic regression

Performance of Bayesian linear regression in a model with mismatch

Low noise sensitivity analysis of -minimization in oversampled systems

Contact Info

Product

Resources

About