Cost-sensitive boosting algorithms: Do we really need them?

Abstract

We provide a unifying perspective for two decades of work on cost-sensitive Boosting algorithms. When analyzing the literature 1997-2016, we find 15 distinct costsensitive variants of the original algorithm; each of these has its own motivation and claims to superiority-so who should we believe? In this work we critique the Boosting literature using four theoretical frameworks: Bayesian decision theory, the functional gradient descent view, margin theory, and probabilistic modelling. Our finding is that only three algorithms are fully supported-and the probabilistic model view suggests that all require their outputs to be calibrated for best performance. Experiments on 18 datasets across 21 degrees of imbalance support the hypothesis-showing that once calibrated, they perform equivalently, and outperform all others. Our final recommendation-based on simplicity, flexibility and Editors: Thomas Gärtner, Mirco Nanni, Andrea Passerini, and Celine Robardet. Learn (2016) 104:359-384 performance-is to use the original Adaboost algorithm with a shifted decision threshold and calibrated probability estimates. Electronic supplementary material

EBNO: Evolution of cost‐sensitive Bayesian networks

“…It also remains an option available in toolkits such as Weka, Scikit learn, and R, so the results should be of interest to practitioners. The original TAN learning algorithm in order to assess the extent to which EBNO makes a difference in comparison to an algorithm that focuses on maximizing accuracy of TANs. Use of two boosting algorithms, AdaBoost to induce cost‐sensitive TANs and use of XGBoost to induce cost‐sensitive decision trees. Although AdaBoost is one of the earliest boosting algorithms, as the study by Nikolaou et al () concludes, it remains an important method for boosting and cost‐sensitive learning. XGBoost is included primarily because it is a more recent innovation and has resulted in some of the best results in Kaggle competitions.…”

“…However, in a more recent paper, Cost‐sensitive boosting algorithms: Do we really need them? , Nikolaou, Narayanan, Kull, Flach, and Brown () present a critique of cost‐sensitive boosting algorithms from multiple perspectives and conclude that Adaboost performs just as well as other variations of boosting algorithms.…”

Tree-Based Cost Sensitive Methods for Fraud Detection in Imbalanced Data

“…The way to transform the output of a boosting model into a probability (the calibration process) plays a key role in the performance of the predictive algorithm. It has been shown that we can achieve at least the same performance with well calibrated boosting model than with one which is cost sensitive [14]. However, we think that our cost sensitive approach gives us a good transformation of the output of the model into a probability.…”

On Boosting: Theory and Applications

“…Additional adaptive boosting algorithm variants like TotalBoost [34], BrownBoost [12], LPBoost [7] and SmoothBoost [33] and those in [8] will not be considered in this tutorial, as well as costsensitive boosting in asymmetric statistical learning problems ( [25], [24] and references therein, in particular those presented in Table 3.1, Chapter 3).…”

“…After a description of AdaBoostClassifier(), we turn briefly our attention to a specific gradient boosting software implementation. In Python, gradient boosting is implemented in the sklearn library by the GradientBoostingClassifier() and GradientBoostingRegressor() functions 25 ; both functions use trees as base learners. Let us focus on classification problems only.…”