Exponential Tail Local Rademacher Complexity Risk Bounds Without the Bernstein Condition

Abstract

The local Rademacher complexity framework is one of the most successful general-purpose toolboxes for establishing sharp excess risk bounds for statistical estimators based on the framework of empirical risk minimization. Applying this toolbox typically requires using the Bernstein condition, which often restricts applicability to convex and proper settings. Recent years have witnessed several examples of problems where optimal statistical performance is only achievable via non-convex and improper estimators originating from aggregation theory, including the fundamental problem of model selection. These examples are currently outside of the reach of the classical localization theory.In this work, we build upon the recent approach to localization via offset Rademacher complexities, for which a general high-probability theory has yet to be established. Our main result is an exponential-tail excess risk bound expressed in terms of the offset Rademacher complexity that yields results at least as sharp as those obtainable via the classical theory. However, our bound applies under an estimator-dependent geometric condition (the "offset condition") instead of the estimator-independent (but, in general, distribution-dependent) Bernstein condition on which the classical theory relies. Our results apply to improper prediction regimes not directly covered by the classical theory.

Distribution-free robust linear regression

“…The following lemma highlights the main property of the Q-aggregation estimator that distinguishes its analysis from the exponential weights. This lemma bears similarity with the analysis using offset Rademacher complexities [45,75,67,38], where the curvature of the loss function allows to obtain a certain "complexity regularizing" term. However, the term obtained in the lemma stated below is different from the one present in the offset Rademacher complexity analysis, which would correspond to a term proportional to…”

Exponential Savings in Agnostic Active Learning Through Abstention

“…Our proof is making most of the standard arguments on the Laplace transform of shifted or offset empirical processes. A similar approach was exploited by many authors Wegkamp (2003); Lecué and Rigollet (2014); Liang, Rakhlin, and Sridharan (2015); Zhivotovskiy and Hanneke (2018); Kanade, Rebeschini, and Vaškevičius (2022) in the statistical setup, though their analysis is specific to strongly convex losses or binary losses under additional probabilistic assumptions. In (Vijaykumar, 2021), the author generalized the approach of Liang et al (2015) to study the properties of ERM under exp-concave losses, though their analysis only focuses on getting the O(1/n) rate of convergence and fails to capture the local structure of the reference set.…”

A Regret-Variance Trade-Off in Online Learning

“…Negative terms typically appear when proving fast rates in statistical learning with squared loss. In particular, the empirical star algorithm of Audibert (2007)-as well as other aggregation algorithms (Lecué and Mendelson, 2009;Lecué and Rigollet, 2014;Wintenberger, 2017)-exploit the curvature of the loss through the negative term which compensates the variance term (see (Kanade et al, 2022) for a detailed discussion in the context of statistical learning). Similarly, in the context of online learning the negative quadratic term appears in (Rakhlin and Sridharan, 2014), where the so-called sequential offset Rademacher complexity is studied.…”