Boosting Algorithms: Regularization, Prediction and Model Fitting

Bühlmann, Peter; Hothorn, Torsten

doi:10.1214/07-sts242

Cited by 750 publications

(934 citation statements)

References 74 publications

Supporting

Mentioning

907

Contrasting

Unclassified

Order By: Relevance

“…The most prominent example is the Lasso (Tibshirani, 1996) and its extensions to Fused Lasso (Tibshirani et al, 2005) and Group Lasso (Yuan & Lin, 2006). Alternative regularized estimators that enforce variable selection are the Elastic Net (Zou & Hastie, 2005), SCAD (Fan & Li, 2001), the Dantzig selector (Candes & Tao, 2007) and boosting approaches (Bühlmann & Yu, 2003;Bühlmann & Hothorn, 2007;Tutz & Binder, 2006). These methods, however, were developed for models with univariate response.…”

Section: Introductionmentioning

confidence: 99%

Variable selection in general multinomial logit models

Tutz

Pößnecker

Uhlmann

2015

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

The use of the multinomial logit model is typically restricted to applications with few predictors, because in high-dimensional settings maximum likelihood estimates tend to deteriorate. In this paper we are proposing a sparsity-inducing penalty that accounts for the special structure of multinomial models. In contrast to existing methods, it penalizes the parameters that are linked to one variable in a grouped way and thus yields variable selection instead of parameter selection. We develop a proximal gradient method that is able to efficiently compute stable estimates. In addition, the penalization is extended to the important case of predictors that vary across response categories. We apply our estimator to the modeling of party choice of voters in Germany including voter-specific variables like age and gender but also party-specific features like stance on nuclear energy and immigration.

show abstract

Section: Introductionmentioning

confidence: 99%

Variable selection in general multinomial logit models

Tutz

Pößnecker

Uhlmann

2015

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

show abstract

“…Instead, this function is updated at each iteration by adding a function, namely B(t) T (m) (x; c), which is smoothed across t according to the penalty. The approach of applying regularization to the base-learner, but not directly to the model itself is common to boosting algorithms, see for instance Büehlmann and Hothorn (2007). Regularization of the boosting model is applied by selecting an appropriate stopping iteration, m stop .…”

Section: Coefficient Boosting With Multivariate Treesmentioning

confidence: 99%

“…Boosting, which originates from machine learning, has become a popular approach for fitting regression and classification models. Overviews are provided by Hastie et al (2009) and Büehlmann and Hothorn (2007). The main application of boosting has been towards estimating models for predicting or classifying a univariate response.…”

mentioning

confidence: 99%

Boosted coefficient models

Sexton¹,

Laake²

2011

Stat Comput

View full text Add to dashboard Cite

Regression methods typically construct a mapping from the covariates into the real numbers. Here, however, we consider regression problems where the task is to form a mapping from the covariates into a set of (univariate) real-valued functions. Examples are given by conditional density estimation, hazard regression and regression with a functional response. Our approach starts by modeling the function of interest using a sum of B-spline basis functions. To model dependence on the covariates, the coefficients of this expansion are each modeled as functions of the covariates. We propose to estimate these coefficient functions using boosted tree models. Algorithms are provided for the above three situations, and real data sets are used to investigate their performance. The results indicate that the proposed methodology performs well. In addition, it is both straightforward, and capable of handling a large number of covariates.

show abstract

“…But even in low-dimensional settings the lack of a clear variable selection strategy provides further challenges. In order to deal with these problems, we propose a new inferential scheme for joint models based on gradient boosting (Bühlmann and Hothorn, 2007). Boosting algorithms emerged from the field of machine learning and were originally designed to enhance the performance of weak classifiers (base-learners) with the aim to yield a perfect discrimination of binary outcomes (Freund and Schapire, 1996).…”

Section: Introductionmentioning

confidence: 99%

Boosting joint models for longitudinal and time‐to‐event data

et al. 2017

View full text Add to dashboard Cite

Joint Models for longitudinal and time-to-event data have gained a lot of attention in the last few years as they are a helpful technique to approach common a data structure in clinical studies where longitudinal outcomes are recorded alongside event times. Those two processes are often linked and the two outcomes should thus be modeled jointly in order to prevent the potential bias introduced by independent modelling. Commonly, joint models are estimated in likelihood based expectation maximization or Bayesian approaches using frameworks where variable selection is problematic and which do not immediately work for high-dimensional data. In this paper, we propose a boosting algorithm tackling these challenges by being able to simultaneously estimate predictors for joint models and automatically select the most influential variables even in high-dimensional data situations. We analyse the performance of the new algorithm in a simulation study and apply it to the Danish cystic fibrosis registry which collects longitudinal lung function data on patients with cystic fibrosis together with data regarding the onset of pulmonary infections. This is the first approach to combine state-of-the art algorithms from the field of machine-learning with the model class of joint models, providing a fully data-driven mechanism to select variables and predictor effects in a unified framework of boosting joint models.

show abstract

Boosting Algorithms: Regularization, Prediction and Model Fitting

Cited by 750 publications

References 74 publications

Variable selection in general multinomial logit models

Variable selection in general multinomial logit models

Boosted coefficient models

Boosting joint models for longitudinal and time‐to‐event data

Contact Info

Product

Resources

About