Fully corrective boosting with arbitrary loss and regularization

Abstract

We propose a general framework for analyzing and developing fully corrective boosting-based classifiers. The framework accepts any convex objective function, and allows any convex (for example, p -norm, p ≥ 1) regularization term. By placing the wide variety of existing fully corrective boosting-based classifiers on a common footing, and considering the primal and dual problems together, the framework allows direct comparison between apparently disparate methods. By solving the primal rather than the dual the framework is capable of generating efficient fully-corrective boosting algorithms without recourse to sophisticated convex optimization processes. We show that a range of additional boosting-based algorithms can be incorporated into the framework despite not being fully corrective. Finally, we provide an empirical analysis of the performance of a variety of the most significant boosting-based classifiers on a few machine learning benchmark datasets.

StructBoost: Boosting Methods for Predicting Structured Output Variables

“…In standard boosting, one has a large number of variables while in SSVM, one has a large number of constraints. For the moment, let us put aside the difficulty of the large number of constraints, and focus on how to iteratively solve for w using column generation as in boosting methods [8], [28], [29]. We derive the Lagrange dual of the optimization of (35) as:…”

“…The subproblem is to add new variables (or constraints for the dual form) into the master problem. With the primal-dual pair of (35) and (37) and following the general framework of column generation based boosting [8], [28], [29], we can obtain our StructBoost as follows: Iterate the following two steps until converge : 1) Solve the following subproblem, which generates the best weak structured learner by finding the most violated constraint in the dual:…”

“…It is well known that, if the following is satisfied for any pair (p, q) ∈ S, the energy function in (22) is submodular: θ (p,q) (0, 0) + θ (p,q) (1, 1) ≤ θ (p,q) (0, 1) + θ (p,q) (1, 0). (24) We want to keep the above property.…”

“…Recall that we use the Hamming loss ∆(y i , y) as defined in (26), the term ∆(y i , y) can be absorbed into the unary term of the energy function defined in (22) (such as in [16]). The inference in (29) can be written as:…”

RBoost: Label Noise-Robust Boosting Algorithm Based on a Nonconvex Loss Function and the Numerically Stable Base Learners

“…It can be interpreted as forward fitting an additive model in functional space by stagewise optimizing the expected risk based on the exponential loss function ϕ e (y f (x)) = exp(−y f (x)) [5], [6]. In particular, AdaBoost maintains a sample distribution D t over the training samples that is initialized from the uniform distribution D t (i ) = 1/m.…”