Distribution-free robust linear regression

Mourtada, Jaouad; Vaškevičius, Tomas; Zhivotovskiy, Nikita

doi:10.4171/msl/27

Cited by 9 publications

(9 citation statements)

References 104 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the recent use of midpoint procedures in statistical literature, see, for example, (Mendelson, 2019;Bousquet and Zhivotovskiy, 2021;Mourtada, Vaškevičius, and Zhivotovskiy, 2022). Since f (mid) outputs 2-sparse convex combinations of elements of the dictionary G, similarly to the above analysis of Audibert's star algorithm, it is enough to establish that f (mid) satisfies the offset condition.…”

Section: Model Selection Aggregationmentioning

confidence: 99%

“…data (e.g., models arising from different tuning parameters, or different statistical estimators) is a fundamental problem in statistics. At the same time, deviation-optimal model selection aggregation procedures have been used to construct computable procedures (not necessarily computationally efficient) to demonstrate the achievability of some statistical minimax lower bounds (see, e.g., (Rakhlin, Sridharan, and Tsybakov, 2017;Mendelson, 2019;Mourtada, Vaškevičius, and Zhivotovskiy, 2022)).…”

Section: Introductionmentioning

confidence: 99%

“…The phenomenon concerning deviation-optimality of model selection aggregation estimators has generated a lot of attention; for example, see the works by Lecué and Mendelson (2009); Rigollet (2012); Dai, Rigollet, and Zhang (2012); Lecué and Rigollet (2014); Wintenberger (2017); Bellec (2017) for analysis of different model selection aggregation procedures. More broadly, the analysis of improper statistical estimators is receiving increased attention, as such procedures were shown to be necessary for optimal statistical performance in logistic regression, see (Hazan, Koren, and Levy, 2014;Foster, Kale, Luo, Mohri, and Sridharan, 2018;Mourtada and Gaïffas, 2019), and linear regression, see (Vaškevičius and Zhivotovskiy, 2020;Mourtada, Vaškevičius, and Zhivotovskiy, 2022).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Kanade¹,

Rebeschini²,

Vaškevičius³

2022

Preprint

View full text Add to dashboard Cite

The local Rademacher complexity framework is one of the most successful general-purpose toolboxes for establishing sharp excess risk bounds for statistical estimators based on the framework of empirical risk minimization. Applying this toolbox typically requires using the Bernstein condition, which often restricts applicability to convex and proper settings. Recent years have witnessed several examples of problems where optimal statistical performance is only achievable via non-convex and improper estimators originating from aggregation theory, including the fundamental problem of model selection. These examples are currently outside of the reach of the classical localization theory.In this work, we build upon the recent approach to localization via offset Rademacher complexities, for which a general high-probability theory has yet to be established. Our main result is an exponential-tail excess risk bound expressed in terms of the offset Rademacher complexity that yields results at least as sharp as those obtainable via the classical theory. However, our bound applies under an estimator-dependent geometric condition (the "offset condition") instead of the estimator-independent (but, in general, distribution-dependent) Bernstein condition on which the classical theory relies. Our results apply to improper prediction regimes not directly covered by the classical theory.

show abstract

Section: Model Selection Aggregationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Kanade¹,

Rebeschini²,

Vaškevičius³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We also show a high-probability excess risk bound for online to batch conversion of an online regression algorithm in the bounded setup, which is to say that both the feature vectors and derivatives of the losses are bounded. It was previously shown by Mourtada et al (2022) that online to batch converted versions of the optimal Vovk-Azoury-Warmuth forecaster (Vovk, 2001;Azoury and Warmuth, 2001) have constant excess risk with constant probability.…”

Section: Contributions and Outlinementioning

confidence: 99%

A Regret-Variance Trade-Off in Online Learning

Hoeven¹,

Zhivotovskiy²,

Cesa-Bianchi³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We consider prediction with expert advice for strongly convex and bounded losses, and investigate trade-offs between regret and "variance" (i.e., squared difference of learner's predictions and best expert predictions). With K experts, the Exponentially Weighted Average (EWA) algorithm is known to achieve O(log K) regret. We prove that a variant of EWA either achieves a negative regret (i.e., the algorithm outperforms the best expert), or guarantees a O(log K) bound on both variance and regret. Building on this result, we show several examples of how variance of predictions can be exploited in learning. In the online to batch analysis, we show that a large empirical variance allows to stop the online to batch conversion early and outperform the risk of the best predictor in the class. We also recover the optimal rate of model selection aggregation when we do not consider early stopping. In online prediction with corrupted losses, we show that the effect of corruption on the regret can be compensated by a large variance. In online selective sampling, we design an algorithm that samples less when the variance is large, while guaranteeing the optimal regret bound in expectation. In online learning with abstention, we use a similar term as the variance to derive the first high-probability O(log K) regret bound in this setting. Finally, we extend our results to the setting of online linear regression.

show abstract

“…For a recent survey with focus on multivariate mean estimation, we refer to [41]. The central ideas behind the robust mean estimation found their applications in many related problems such as regression [28,9,43,19,46,52], covariance estimation [15,16,49,53,26,51], and clustering [33]. For related results in the context of covariance estimation for heavy-tailed distributions, we refer to the recent survey [32].…”

Section: Introductionmentioning

confidence: 99%

Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tails

Abdalla¹,

Zhivotovskiy²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We provide an estimator of the covariance matrix that achieves the optimal rate of convergence (up to constant factors) in the operator norm under two standard notions of data contamination: We allow the adversary to corrupt an η-fraction of the sample arbitrarily, while the distribution of the remaining data points only satisfies that the L pmarginal moment with some p 4 is equivalent to the corresponding L 2 -marginal moment. Despite requiring the existence of only a few moments, our estimator achieves the same tail estimates as if the underlying distribution were Gaussian. As a part of our analysis, we prove a dimension-free Bai-Yin type theorem in the regime p > 4.

show abstract

Distribution-free robust linear regression

Cited by 9 publications

References 104 publications

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

A Regret-Variance Trade-Off in Online Learning

Covariance Estimation: Optimal Dimension-free Guarantees for Adversarial Corruption and Heavy Tails

Contact Info

Product

Resources

About