Concentration Inequalities

Abstract

Concentration inequalities deal with deviations of functions of independent random variables from their expectation. In the last decade new tools have been introduced making it possible to establish simple and powerful inequalities. These inequalities are at the heart of the mathematical analysis of various problems in machine learning and made it possible to derive new efficient algorithms. This text attempts to summarize some of the basic tools.

On aggregation for heavy-tailed classes

“…Fix t j to be named later and apply Talagrand's concentration inequality for bounded empirical processes [24,14,4] to the class of indicator functions {½ {|f −πf |≥t j } : f ∈ F}. Thus, with probability at least 1 − 2 exp(−m), for every f ∈ F,…”

“…By the bounded differences inequality (see, for example, [4]), with probability at least 1 − exp(−c 7 t 2 ),…”

“…Fortunately, there are highly potent tools at one's disposal when bounded classes are concerned, namely, Talagrand's concentration inequality for bounded empirical processes and the contraction principle for empirical and Bernoulli processes indexed by bounded classes (see, e.g., [15,26,4]). Using that well established machinery, one may show that A u 0 is a high probability event.…”

Lower Bounds on the Generalized Central Moments of the Optimal Alignments Score of Random Sequences

“…As shown in [16], EfronStein type inequalities can be applied to Φ(x) = |x| r , r > 0, via the Burkholder's inequality, giving the aforementioned constant E(r). Another generalization of the Efron-Stein inequality, given, for example, in [6,Chapter 14], is the subadditivity inequality of the so called Φ-Entropy. However, it only applies to those functions Φ having strictly positive second derivative and such that 1/Φ ′′ is concave, e.g., Φ(x) = |x| r , for r ∈ (1, 2], or Φ(x) = x log x.…”

“…However, it only applies to those functions Φ having strictly positive second derivative and such that 1/Φ ′′ is concave, e.g., Φ(x) = |x| r , for r ∈ (1, 2], or Φ(x) = x log x. Moreover, the upper bounds on (3.1) for those Φ are linear in n as easily seen via, e.g., [6,Theorem 14.6], together with (2.1). In [6,Chapter 15], some further generalizations of Efron-stein inequality for Φ(x) = |x| r , r ≥ 2, are also obtained, which, together with (2.1), also imply upper bounds of order n r/2 on the r-th central moments (cf.…”