Component-wise AdaBoost algorithms for high-dimensional binary classification and class probability prediction

Abstract

Freund and Schapire (1997) introduced "Discrete AdaBoost"(DAB) which has been mysteriously effective for the high-dimensional binary classification or binary prediction. In an effort to understand the myth, Friedman, Hastie and Tibshirani (FHT, 2000) show that DAB can be understood as statistical learning which builds an additive logistic regression model via Newton-like updating minimization of the"exponential loss". From this statistical point of view, FHT proposed three modifications of DAB, namely, Real AdaBoost (RAB), LogitBoost (LB), and Gentle AdaBoost (GAB). All of DAB, RAB, LB, GAB solve for the logistic regression via different algorithmic designs and different objective functions. The RAB algorithm uses class probability estimates to construct real-valued contributions of the weak learner, LB is an adaptive Newton algorithm by stagewise optimization of the Bernoulli likelihood, and GAB is an adaptive Newton algorithm via stagewise optimization of the exponential loss. The same authors of FHT published an influential textbook, The Elements of Statistical Learning (ESL, 2001 and 2008). A companion book An Introduction to Statistical Learning (ISL) by James et al. (2013) was published with applications in R. However, both ESL and ISL (e.g., sections 4.5 and 4.6) do not cover these four AdaBoost algorithms while FHT provided some simulation and empirical studies to compare these methods. Given numerous potential applications, we believe it would be useful to collect the R libraries of these AdaBoost algorithms, as well as more recently developed extensions to Ad-aBoost for probability prediction with examples and illustrations. Therefore, the goal of this chapter is to do just that, i.e., (i) to provide a user guide of these alternative AdaBoost algorithms with step-by-step tutorial of using R (in a way similar to ISL, e.g., Section 4.6), (ii) to compare AdaBoost with alternative machine learning classification tools such as the deep neural network (DNN), logistic regression with LASSO

Whether pattern memory can be truly realized in deep neural network?

“…Although interpretable technologies are emerging in various domains, achieving transparent, reliable and highly generalizable DNN models remains challenging [27][28][29][30][31][32][33][34][35] . Especially when confronted with extensive and intricate data, even the cutting-edge DNN models [36][37][38][39][40][41] utilizing learning techniques for deep feature patterns [42][43][44][45][46] , are unable to achieve unlimited perceptual capabilities [47][48][49] . In recent years, numerous studies have utilized visual illusions to investigate the perceptual neural mechanisms of human visual as well as the performance deficiencies of DNN models [50][51][52][53][54][55][56][57] , which provide valuable insights for narrowing gaps between AI capability and the extraordinary perceptual ability of human brain.…”