The generalization performance of ERM algorithm with strongly mixing observations

Abstract

The generalization performance is the main concern of machine learning theoretical research. The previous main bounds describing the generalization ability of the Empirical Risk Minimization (ERM) algorithm are based on independent and identically distributed (i.i.d.) samples. In order to study the generalization performance of the ERM algorithm with dependent observations, we first establish the exponential bound on the rate of relative uniform convergence of the ERM algorithm with exponentially strongly mixing observations, and then we obtain the generalization bounds and prove that the ERM algorithm with exponentially strongly mixing observations is consistent. The main results obtained in this paper not only extend the previously known results for i.i.d. observations to the case of exponentially strongly mixing observations, but also improve the previous results for strongly mixing samples. Because the ERM algorithm is usually very time-consuming and overfitting may happen when the complexity of the hypothesis space is high, as an application of our main results we also explore a new strategy to implement the ERM algorithm in high complexity hypothesis space.

Robustness and generalization

“…Extension, if possible, to non-standard setups-setups where either data are not IID or the minimizing objective is not the expected loss-often requires specifically tailored analysis (e.g. Gamarnik 2003;Lozano et al 2006;Zou et al 2009). In contrast, extension of robustness-based analysis to non-standard setups is straightforward.…”

Learning from uniformly ergodic Markov chains

“…That is, we estimate the quantity (for any ε > 0) (11) by following the enlightening idea of [13,20]. Such a study is motivated by the observation that a uniform convergence bound fails to capture the phenomenon that for those functions f ∈ F for which the expected risk E (f ) is small, the deviation E(f )−E n (f ) is also small with large probability.…”

“…We denote by I 1 the quantity on the right-hand side of inequality (15), and by I 2 the quantity on the left-hand side of inequality (15). Then, take δ = ε E (f j ) , suppose that 0 < ε ≤ 2 3 (aL), and use the similar method of [13], we have…”

“…Zou and Li [12] established the bounds on the rate of uniform convergence of learning machines with strongly mixing observations. Zou, Li and Xu [13] obtained the generalization bounds of the ERM algorithm with exponentially strongly mixing observations.…”

Generalization bounds of ERM algorithm with V-geometrically Ergodic Markov chains

“…Smale and Zhou [14] considered online learning algorithm based on Markov sampling. Zou et al [26] established the bounds on the generalization performance of the ERM algorithm with strongly mixing observations. Xu and Chen [22] considered the learning rates of regularized regression algorithm with strongly mixing sequences.…”