An Introduction to Boosting and Leveraging

Abstract

Abstract. We provide an introduction to theoretical and practical aspects of Boosting and Ensemble learning, providing a useful reference for researchers in the field of Boosting as well as for those seeking to enter this fascinating area of research. We begin with a short background concerning the necessary learning theoretical foundations of weak learners and their linear combinations. We then point out the useful connection between Boosting and the Theory of Optimization, which facilitates the understanding of Boosting and later on enables us to move on to new Boosting algorithms, applicable to a broad spectrum of problems. In order to increase the relevance of the paper to practitioners, we have added remarks, pseudo code, "tricks of the trade", and algorithmic considerations where appropriate. Finally, we illustrate the usefulness of Boosting algorithms by giving an overview of some existing applications. The main ideas are illustrated on the problem of binary classification, although several extensions are discussed.

On the equivalence of weak learnability and linear separability: new relaxations and efficient boosting algorithms

“…For an overview see for example [FS99,Sch03,MR03]. The first boosting algorithm was used for showing the equivalence between weak learnability and strong learnability [Sch90].…”

Weighting machine learning solutions by economic and institutional context for decision making

“…One of the first model assembly systems was bagging, proposed by Breiman (1996) and Buhlmann and Yu (2002) and implemented in R by Spanish researchers, Alfaro et al (2013). Among the resampling techniques, we can found boosting, in particular, the algorithm AdaBoost M1, introduced by Freund and Schapire (1997) and extensively assessed in many studies and analyses, most notably by Eibl and Pfeiffer (2002) and Meir and Rätsch (2003). Random forests is another resampling method developed by Breiman (2001) as a variant of the bagging methodology using decision trees.…”

Precise statements of convergence for AdaBoost and arc-gv

“…Breiman's arc-gv is quite similar to AdaBoost (in fact the pseudocodes differ by only one line), though AdaBoost has been found to exhibit interesting dynamical behavior that may sometimes resemble chaos, or may sometimes converge to provably stable cycles [13] (one can now imagine why AdaBoost is difficult to analyze) whereas arc-gv converges very nicely. See [17] or [7] for an introduction to boosting.…”