On aggregation for heavy-tailed classes

Abstract

We introduce an alternative to the notion of 'fast rate' in Learning Theory, which coincides with the optimal error rate when the given class happens to be convex and regular in some sense. While it is well known that such a rate cannot always be attained by a learning procedure (i.e., a procedure that selects a function in the given class), we introduce an aggregation procedure that attains that rate under rather minimal assumptions -for example, that the L q and L 2 norms are equivalent on the linear span of the class for some q > 2, and the target random variable is square-integrable.

Regularization, sparse recovery, and median-of-means tournaments

“…where Med m (w) is a median of the n values 1 m i∈I j |(f − h)(X i )|. The behaviour of Φ described below has been established in Mendelson [14] (see also Lugosi and Mendelson [10]): • For 1 ≤ ℓ ≤ K and ρ > 0, let r > r * (F, F ℓ , ρ), where r * is defined relative to the constants κ and η).…”

Distribution-Free Robust Linear Regression

“…The main difference with the latter work is that our truncated class is now non-convex, and to obtain the correct rates of convergence, we need to adapt the arguments used in the model selection aggregation literature. This motivates our fourth step that can be seen as an adaptation of the star algorithm [2] and the two-step aggregation procedure developed in [40,51] to our specific heavy-tailed setting combined with the idea of min-max formulation of robust estimators [5,39]. We remark that the idea of combining model selection aggregation techniques with the median-of-means tournaments has also recently appeared in [52], but under different assumptions.…”

“…This fact can be established using the recent result of Shamir [66,Theorem 3], and it remains true even when d = 1 and the response variable Y is bounded almost surely. This observation separates our setup from the existing literature where only proper estimators are studied for convex classes such as F lin even in the heavy-tailed scenarios (see, for example, [14,45,51,52]).…”

“…Moreover, we show in Section 6 that Assumption 1 is necessary to obtain any non-trivial guarantee in our setup. The estimator of Theorem C naturally leverages the ideas of the analysis of truncated linear functions [28,Chapter 11], skeleton estimators [21,Section 28.3], the deviation optimal model selection aggregation procedures [2,40,51], min-max estimators [5,39], and the median-of-means tournaments [45]. An extended discussion is deferred to Section 5.…”

On Monte-Carlo methods in convex stochastic optimization

“…The current focus in statistical learning literature is on non-asymptotic statements that hold under increasingly relaxed assumptions. Among the outcomes of this approach were [20,23,24]-alternatives to the sample average approximation in statistical learning problems that recover the Gaussian rates in completely heavy tailed situations. We believe that pursuing the same direction in the context of stochastic optimization would lead to intriguing questions.…”