Precise statements of convergence for AdaBoost and arc-gv

We introduce a useful tool for analyzing boosting algorithms called the ``smooth margin function,'' a differentiable approximation of the usual margin for boosting algorithms. We present two boosting algorithms based on this smooth margin, ``coordinate ascent boosting'' and ``approximate coordinate ascent boosting,'' which are similar to Freund and Schapire's AdaBoost algorithm and Breiman's arc-gv algorithm. We give convergence rates to the maximum margin solution for both of our algorithms and for arc-gv. We then study AdaBoost's convergence properties using the smooth margin function. We precisely bound the margin attained by AdaBoost when the edges of the weak classifiers fall within a specified range. This shows that a previous bound proved by R\"{a}tsch and Warmuth is exactly tight. Furthermore, we use the smooth margin to capture explicit properties of AdaBoost in cases where cyclic behavior occurs.Comment: Published in at http://dx.doi.org/10.1214/009053607000000785 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org

show abstract

“…Sections 8, 9 and 10 contain proofs from Sections 3, 5, 6 and 7. Preliminary and less detailed statements of these results appear in [25,26].…”

Section: Convergence Properties Of New and Old Algorithmsmentioning

confidence: 94%

Analysis of boosting algorithms using the smooth margin function

Rudin

Schapire²,

Daubechies

2007

Ann. Statist.

Self Cite

View full text Add to dashboard Cite

show abstract

“…But it still has its downsides, as Breiman's quote above indicates. There is evidence both for and against the power of the margin theory to predict the quality of the generalization performance [Breiman, 1999, Rudin et al, 2004, 2007a,b, Reyzin and Schapire, 2006. But the most striking problem is that the margin bound is very loose: It does not explain the precise behavior of the error.…”

Section: The Margin Theorymentioning

confidence: 99%

On the Convergence Properties of Optimal AdaBoost

Belanich¹,

Ortiz²

2012

Preprint

View full text Add to dashboard Cite

AdaBoost is one of the most popular machine-learning algorithms. It is simple to implement and often found very effective by practitioners, while still being mathematically elegant and theoretically sound. AdaBoost's behavior in practice, and in particular the test-error behavior, has puzzled many eminent researchers for over a decade: It seems to defy our general intuition in machine learning regarding the fundamental trade-off between model complexity and generalization performance. In this paper, we establish the convergence of "Optimal AdaBoost," a term coined by Rudin, Daubechies, and Schapire in 2004. We prove the convergence, with the number of rounds, of the classifier itself, its generalization error, and its resulting margins for fixed data sets, under certain reasonable conditions. More generally, we prove that the time/per-round average of almost any function of the example weights converges. Our approach is to frame AdaBoost as a dynamical system, to provide sufficient conditions for the existence of an invariant measure, and to employ tools from ergodic theory. Unlike previous work, we do not assume AdaBoost cycles; actually, we present empirical evidence against it on real-world datasets. Our main theoretical results hold under a weaker condition. We show sufficient empirical evidence that Optimal AdaBoost always met the condition on every real-world dataset we tried. Our results formally ground future convergencerate analyses, and may even provide opportunities for slight algorithmic modifications to optimize the generalization ability of AdaBoost classifiers, thus reducing a practitioner's burden of deciding how long to run the algorithm.

show abstract

Approximation Stability and Boosting

Gao

Zhou

2010

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Precise statements of convergence for AdaBoost and arc-gv

Cited by 8 publications

References 12 publications

Analysis of boosting algorithms using the smooth margin function

Analysis of boosting algorithms using the smooth margin function

On the Convergence Properties of Optimal AdaBoost

Approximation Stability and Boosting

Contact Info

Product

Resources

About