Improved boosting algorithms using confidence-rated predictions

Abstract

We describe several improvements to Freund and Schapire's AdaBoost boosting algorithm, particularly in a setting in which hypotheses may assign confidences to each of their predictions. We give a simplified analysis of AdaBoost in this setting, and we show how this analysis can be used to find improved parameter settings as well as a refined criterion for training weak hypotheses. We give a specific method for assigning confidences to the predictions of decision trees, a method closely related to one used by Quinlan. This method also suggests a technique for growing decision trees which turns out to be identical to one proposed by Keams and Mansour.We focus next on how to apply the new boosting algorithms to multiclass classification problems, paaicularly to the multi-label case in which each example may belong to more than one class. We give two boosting methods for this problem. One of these leads to a new method for handling the single-label case which is simpler but as effective as techniques suggested by Freund and Schapire. Finally, we give some experimental results comparing a few of the algorithms discussed in this paper.

Fully corrective boosting with arbitrary loss and regularization

“…Here we assume a discrete AdaBoost. It is straightforward to extend to real-valued h(·) ∈ [−1, +1] as discussed in [30].…”

Exploiting diversity of margin-based classifiers

“…The purpose of boosting algorithms is to find a highly accurate classification rule by combining many weak or base classifiers. In this work we use the generalized AdaBoost algorithm presented in [13] by Schapire and Singer.…”

“…The learning bias of AdaBoost is proven to be very aggressive at maximizing the margin of the training examples and this makes a clear connection to the SVM learning paradigm [13]. More details about the relation between AdaBoost and SVM can be found in [8], [12].…”

“…The leaves of a tree define a partition over the input space X , and each leaf contains the prediction of the tree for the corresponding part of X . We follow the criterion presented in [13] for growing base decision trees and computing the predictions in the leaves. A maximum depth is used as the stopping criterion.…”

“…Typically, the inductive bias of a margin-based learning method allows to consider the output values of a classifier as a good measure of the confidence on its predictions. In this work, several comparisons among Support Vector Machines (SVM) [15], AdaBoost [13] and a recently proposed model for maximizing the margin with Feed-forward Neural Networks (FNN) [9] have been made. The empirical study has been performed on the Text Categorization task, a real-world classification problem from the Information Retrieval domain, and extends initial experimentation involving the aforementioned algorithms [10].…”