Exponential Savings in Agnostic Active Learning Through Abstention

Puchkin, Nikita; Zhivotovskiy, Nikita

doi:10.1109/tit.2022.3156592

Cited by 5 publications

(2 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since f (mid) outputs 2-sparse convex combinations of elements of the dictionary G, similarly to the above analysis of Audibert's star algorithm, it is enough to establish that f (mid) satisfies the offset condition. For the midpoint estimator, this fact is already implicit in the proofs of Puchkin and Zhivotovskiy (2021) in the context of active learning. While, admittedly, the direct analysis of the midpoint estimator is no more difficult than establishing the below lemma, for exposition purposes, let us demonstrate that f (min) does indeed satisfy the offset condition.…”

Section: Model Selection Aggregationmentioning

confidence: 95%

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Kanade¹,

Rebeschini²,

Vaškevičius³

2022

Preprint

View full text Add to dashboard Cite

The local Rademacher complexity framework is one of the most successful general-purpose toolboxes for establishing sharp excess risk bounds for statistical estimators based on the framework of empirical risk minimization. Applying this toolbox typically requires using the Bernstein condition, which often restricts applicability to convex and proper settings. Recent years have witnessed several examples of problems where optimal statistical performance is only achievable via non-convex and improper estimators originating from aggregation theory, including the fundamental problem of model selection. These examples are currently outside of the reach of the classical localization theory.In this work, we build upon the recent approach to localization via offset Rademacher complexities, for which a general high-probability theory has yet to be established. Our main result is an exponential-tail excess risk bound expressed in terms of the offset Rademacher complexity that yields results at least as sharp as those obtainable via the classical theory. However, our bound applies under an estimator-dependent geometric condition (the "offset condition") instead of the estimator-independent (but, in general, distribution-dependent) Bernstein condition on which the classical theory relies. Our results apply to improper prediction regimes not directly covered by the classical theory.

show abstract

Section: Model Selection Aggregationmentioning

confidence: 95%

Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Kanade¹,

Rebeschini²,

Vaškevičius³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Exploiting negative regret. The possibility of getting a negative excess risk has been recently explicitly exploited by Puchkin and Zhivotovskiy (2022) in the setup of active learning with abstentions. In the context of online to batch conversion of online learning algorithms, a similar idea is exploited in Section 4.1.…”

Section: Related Workmentioning

confidence: 99%

A Regret-Variance Trade-Off in Online Learning

Hoeven¹,

Zhivotovskiy²,

Cesa-Bianchi³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

We consider prediction with expert advice for strongly convex and bounded losses, and investigate trade-offs between regret and "variance" (i.e., squared difference of learner's predictions and best expert predictions). With K experts, the Exponentially Weighted Average (EWA) algorithm is known to achieve O(log K) regret. We prove that a variant of EWA either achieves a negative regret (i.e., the algorithm outperforms the best expert), or guarantees a O(log K) bound on both variance and regret. Building on this result, we show several examples of how variance of predictions can be exploited in learning. In the online to batch analysis, we show that a large empirical variance allows to stop the online to batch conversion early and outperform the risk of the best predictor in the class. We also recover the optimal rate of model selection aggregation when we do not consider early stopping. In online prediction with corrupted losses, we show that the effect of corruption on the regret can be compensated by a large variance. In online selective sampling, we design an algorithm that samples less when the variance is large, while guaranteeing the optimal regret bound in expectation. In online learning with abstention, we use a similar term as the variance to derive the first high-probability O(log K) regret bound in this setting. Finally, we extend our results to the setting of online linear regression.

show abstract