Robust statistical learning with Lipschitz and convex loss functions

Abstract

We obtain estimation and excess risk bounds for Empirical Risk Minimizers (ERM) and minmax Median-Of-Means (MOM) estimators based on loss functions that are both Lipschitz and convex. Results for the ERM are derived under weak assumptions on the outputs and subgaussian assumptions on the design as in [2]. The difference with [2] is that the global Bernstein condition of this paper is relaxed here into a local assumption. We also obtain estimation and excess risk bounds for minmax MOM estimators under similar assumptions on the output and only moment assumptions on the design. Moreover, the dataset may also contains outliers in both inputs and outputs variables without deteriorating the performance of the minmax MOM estimators.Unlike alternatives based on MOM's principle [24,29], the analysis of minmax MOM estimators is not based on the small ball assumption (SBA) of [22]. In particular, the basic example of non parametric statistics where the learning class is the linear span of localized bases, that does not satisfy SBA [39] can now be handled. Finally, minmax MOM estimators are analysed in a setting where the local Bernstein condition is also dropped out. It is shown to achieve excess risk bounds with exponentially large probability under minimal assumptions insuring only the existence of all objects. * • The hinge loss defined, for any u ∈Ȳ = R and y ∈ Y = {−1, 1}, by¯ (u, y) = max(1 − uy, 0) satisfies Assumption 1 with L = 1.• The Huber loss defined, for anysatisfies Assumption 1 with L = δ.• The quantile loss is defined, for any τ. It satisfies Assumption 1 with L = 1. For τ = 1/2, the quantile loss is the L 1 loss.All along the paper, the following assumption is also granted.Assumption 2. The class F is convex.The empirical risk minimizers (ERM) [43] obtained by minimizing f ∈ F → R N (f ) are expected to be close to the oracle f * . This procedure and its regularized versions have been extensively studied in learning theory [20]. When the loss is both convex and Lipschitz, results have been obtained in practice [4,12] and theory [42]. Risk bounds with exponential deviation inequalities for the ERM can be obtained under weak assumptions on the outputs Y , but stronger assumptions on the design X. Moreover, fast rates of convergence [41] can only be obtained under margin type assumptions such as the Bernstein condition [8,42].The Lipschitz assumption and global Bernstein conditions (that hold over the entire F as in [2]) imply boundedness in L 2 -norm of the class F , see the discussion preceding Assumption 4 for details. This boundedness is not satisfied in linear regression with unbounded design so the results of [2] don't apply to this basic example such as linear regression with a Gaussian design. To bypass this restriction, the global condition is relaxed into a "local" one as in [15,42], see Assumption 4 below.The main constraint in our results on ERM is the assumption on the design. This constraint can be relaxed by considering alternative estimators based on the "median-of-means" (MOM) principle of [...

ERM and RERM are optimal estimators for regression problems when malicious outliers corrupt the labels

“…Regularized empirical risk minimization is the most widespread strategy in machine learning to estimate f * . There exists an extensive literature on its generalization capabilities [56,31,30,35,18]. However, in the past few years, many papers have highlighted its severe limitations.…”

“…Based on the previous works [18,16,17, 1], we study both ERM and RERM for regression problems when the penalization is a norm and the loss function is simultaneously convex and Lipschitz (Assumption 3) and show that:…”

“…The results are derived under the local Bernstein condition introduced in [18]. This condition enables to obtain fast rates of convergence, even when the noise is heavy-tailed.…”

Iteratively reweighted ℓ1-penalized robust regression

“…For more complex problems in which both the covariates and noise can be (i) heavy-tailed and/or (ii) adversarially contaminated, the estimator obtained by minimizing a robust loss function is still sensitive to outliers in the feature space. To achieve robustness in both feature and response spaces, recent years have witnessed a rapid development of the "median-of-means" (MOM) principle, which dates back to [40] and [22], and a variety of MOM-based procedures for regression and classification in both low-an high-dimensional settings [12,13,26,27,33,35]. We refer to [34] for a recent survey.…”

“…Note that Condition 4.1 excludes some important Lipschitz continuous functions, such as the check function for quantile regression and the hinge loss for classification, which do not have a local strong convexity. The recent works [1] and [12,13] established optimal estimation and excess risk bounds for (regularized) empirical risk minimizers and MOM-type estimators based on general convex and Lipschitz loss functions even without a local quadratic behavior. Our work complements the existing results on 1 -regularized ERM by showing oracle properties of nonconvex regularized methods under stronger signals.…”