Excess risk bounds in robust empirical risk minimization

Abstract

This paper investigates robust versions of the general empirical risk minimization algorithm, one of the core techniques underlying modern statistical methods. Success of the empirical risk minimization is based on the fact that for a ‘well-behaved’ stochastic process $\left \{ f(X), \ f\in \mathscr F\right \}$ indexed by a class of functions $f\in \mathscr F$, averages $\frac{1}{N}\sum _{j=1}^N f(X_j)$ evaluated over a sample $X_1,\ldots ,X_N$ of i.i.d. copies of $X$ provide good approximation to the expectations $\mathbb E f(X)$, uniformly over large classes $f\in \mathscr F$. However, this might no longer be true if the marginal distributions of the process are heavy tailed or if the sample contains outliers. We propose a version of empirical risk minimization based on the idea of replacing sample averages by robust proxies of the expectations and obtain high-confidence bounds for the excess risk of resulting estimators. In particular, we show that the excess risk of robust estimators can converge to $0$ at fast rates with respect to the sample size $N$, referring to the rates faster than $N^{-1/2}$. We discuss implications of the main results to the linear and logistic regression problems and evaluate the numerical performance of proposed methods on simulated and real data.

Convergence Rates for Learning Linear Operators from Noisy Data

“…The first problem in (2.13) is known as empirical risk minimization [41,56]; the second is its regularized counterpart, and its minimizer, denoted by L(N,R) , is the focus of our discussion. These problems are meant to approximate the best linear estimator given infinite data, denoted by L, which is a solution to inf{R ∞ (L) : L linear and measurable}; this agrees with solutions…”

“…In our infinite-dimensional setting, it is also interesting to see from Theorem 3.7 that so-called "fast rates" for the excess risk (i.e., faster than N −1/2 , see [41]) may be attained by the posterior operator estimator in certain parameter regimes. The usual statistical learning theory techniques based on bounding suprema of empirical processes typically yield "slow" N −1/2 rates or worse (see, e.g., [56]).…”

On Catoni's M-Estimation

“…Other robust mean estimators might also be employed to get another robust version of empirical risk minimization. For example, Mathieu and Minsker [10] used Minsker's estimator [11] for robust empirical risk minimization and gave high-confidence bounds for excessive risk (see Mathieu and Minsker [10] for details).…”

Uniform Concentration Bounds toward a Unified Framework for Robust Clustering

“…MoM estimators are not only insensitive to outliers, but are also equipped with exponential concentration results under the mild condition of finite variance (Lugosi and Mendelson, 2019;Lerasle, 2019;Laforgue et al, 2019). Recently, near-optimal results for mean estimation (Minsker, 2018), classification , regression (Mathieu and Minsker, 2021;Lugosi and Mendelson, 2019), clustering (Klochkov et al, 2020;Brunet-Saumard et al, 2022), bandits (Bubeck et al, 2013) and optimal transport (Staerman et al, 2021) have been established from this perspective.…”