Totally corrective boosting algorithms that maximize the margin

Abstract

We consider boosting algorithms that maintain a distribution over a set of examples. At each iteration a weak hypothesis is received and the distribution is updated. We motivate these updates as minimizing the relative entropy subject to linear constraints. For example AdaBoost constrains the edge of the last hypothesis w.r.t. the updated distribution to be at most γ = 0. In some sense, AdaBoost is "corrective" w.r.t. the last hypothesis. A cleaner boosting method is to be "totally corrective": the edges of all past hypotheses are constrained to be at most γ, where γ is suitably adapted.Using new techniques, we prove the same iteration bounds for the totally corrective algorithms as for their corrective versions. Moreover with adaptive γ, the algorithms provably maximizes the margin. Experimentally, the totally corrective versions return smaller convex combinations of weak hypotheses than the corrective ones and are competitive with LPBoost, a totally corrective boosting algorithm with no regularization, for which there is no iteration bound known.

Fully corrective boosting with arbitrary loss and regularization

“…It uses coordinate descent, however, and is therefore not fully (totally) corrective. Totally corrective boosting algorithms, like LPBoost [3], TotalBoost [18] and those proposed in [13], update the coefficients of all previously selected weak learners at each iteration. The fully corrective boosting algorithms thus require significantly fewer training iterations to achieve convergence [13] and result in smaller, and therefore more efficient, ensemble classifiers.…”

“…In the proposed CGBoost, generally we do not require sophisticated convex solvers and only gradient descent methods like L-BFGS-B [19] are needed. Previous totally-corrective boosting algorithms [3,13,18] all solve the dual problems using convex optimization solvers. Besides the primal problems' much simpler structures, in most cases, the dual problems have many more variables than their corresponding primal problems.…”

“…L-BFGS-B is faster and more scalable. TotalBoost [18] takes the same form as (17) except that the parameter ϑ is adaptively set to the minimum edge over all 9 weak classifiers generated up to the current iteration. It is clear now that TotalBoost is also a regularized AdaBoost:…”

“…AdaBoost * ϑ [29] and TotalBoost [18] minimize the regularized versions of AdaBoost's loss with an adaptive regularization parameter, which is the minimum edge over all weak classifiers (up to a numerical accuracy ϑ).…”

“…TotalBoost [18] failed to discuss the primal-dual relationship of AdaBoost's cost function and (17), and an LPBoost is solved to obtain the final strong classifier after iteratively solving (17). In other words, in TotalBoost, the dual problem (17) is only used to generate weak classifiers.…”