Learning without Concentration

Abstract

We obtain sharp bounds on the performance of Empirical Risk Minimization performed in a convex class and with respect to the squared loss, without assuming that class members and the target are bounded functions or have rapidly decaying tails.Rather than resorting to a concentration-based argument, the method used here relies on a 'small-ball' assumption and thus holds for classes consisting of heavy-tailed functions and for heavy-tailed targets.The resulting estimates scale correctly with the 'noise level' of the problem, and when applied to the classical, bounded scenario, always improve the known bounds.

On aggregation for heavy-tailed classes

“…We will outline the essentials of this method in what follows, but refer the reader to [18,19] for a more detailed description of the parameters involved, their role in the analysis of ERM and the way in which they may be computed in specific applications.…”

“…The third and final complexity parameter is also a minor modification of a similar parameter from [18,19]. It will be used to study the multiplier component in the decomposition (1.2) of the excess squared-loss functional.…”

“…for an arbitrary class F. The small-ball method, introduced in [18,17,9,19], may be used to show that if F satisfies the small-ball condition with constants κ 0 and ε then inf…”

“…The obvious downside in this concentration-contraction based argument is that it totally eliminates the dependence on the distance between F and Y (see the discussion in [18,19] for more details). As an outcome, the estimate in the bounded case does not improve when the problem becomes more 'realizable'.…”

“Local” vs. “global” parameters—breaking the Gaussian complexity barrier

“…This question has been studied extensively, and we refer the reader to the manuscripts [2,9,6,17,3,3,10,11] for more information on its history and on some more recent progress.…”