Gradient boosting for distributional regression: faster tuning and improved variable selection via noncyclical updates

Thomas, Janek; Mayr, Andreas; Bischl, Bernd; Schmid, Matthias; Smith, Adam D.; Hofner, Benjamin

doi:10.1007/s11222-017-9754-6

Cited by 67 publications

(91 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, this framework allowed us to incorporate smooth effects to efficiently account for spatiotemporal trends in the data that were poorly explained by other covariates and to identify those covariates and their functional forms most consistently associated with animal distribution and abundance (via stability selection). Recent advances in the application of gradient boosting (non‐cyclical application: see Thomas et al, ) could allow for even greater power in selecting appropriate variables and responses from among available covariates.…”

Section: Discussionmentioning

confidence: 99%

“…We fitted GAM and GAMLSS using an iterative machine‐learning approach, component‐wise functional gradient descent boosting (Bühlmann & Hothorn, ; Hothorn et al, ; Mayr et al, ; Hofner, Boccuto, & Göker, ; Mayr & Hofner, ) in a cyclical framework (Thomas et al, ). The first step of this process was to compute the negative gradient of a pre‐selected loss function, which acts as a working residual by giving more weight to observations not properly predicted in previous iterations.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Modeling spatiotemporal abundance of mobile wildlife in highly variable environments using boosted GAMLSS hurdle models

Smith

Hofner

Lamb

et al. 2019

Ecology and Evolution

Self Cite

View full text Add to dashboard Cite

Modeling organism distributions from survey data involves numerous statistical challenges, including accounting for zero‐inflation, overdispersion, and selection and incorporation of environmental covariates. In environments with high spatial and temporal variability, addressing these challenges often requires numerous assumptions regarding organism distributions and their relationships to biophysical features. These assumptions may limit the resolution or accuracy of predictions resulting from survey‐based distribution models. We propose an iterative modeling approach that incorporates a negative binomial hurdle, followed by modeling of the relationship of organism distribution and abundance to environmental covariates using generalized additive models (GAM) and generalized additive models for location, scale, and shape (GAMLSS). Our approach accounts for key features of survey data by separating binary (presence‐absence) from count (abundance) data, separately modeling the mean and dispersion of count data, and incorporating selection of appropriate covariates and response functions from a suite of potential covariates while avoiding overfitting. We apply our modeling approach to surveys of sea duck abundance and distribution in Nantucket Sound (Massachusetts, USA), which has been proposed as a location for offshore wind energy development. Our model results highlight the importance of spatiotemporal variation in this system, as well as identifying key habitat features including distance to shore, sediment grain size, and seafloor topographic variation. Our work provides a powerful, flexible, and highly repeatable modeling framework with minimal assumptions that can be broadly applied to the modeling of survey data with high spatiotemporal variability. Applying GAMLSS models to the count portion of survey data allows us to incorporate potential overdispersion, which can dramatically affect model results in highly dynamic systems. Our approach is particularly relevant to systems in which little a priori knowledge is available regarding relationships between organism distributions and biophysical features, since it incorporates simultaneous selection of covariates and their functional relationships with organism responses.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Modeling spatiotemporal abundance of mobile wildlife in highly variable environments using boosted GAMLSS hurdle models

Smith

Hofner

Lamb

et al. 2019

Ecology and Evolution

Self Cite

View full text Add to dashboard Cite

show abstract

“…Note that updating scheme (8) can also be used as a weak base learner to build a component-wise gradient boosting algorithm (Mayr et al, 2012;Thomas et al, 2018) for the fusion penalties presented in Section 3. This technique has the advantage that each model term X jk β jk can have a different amount of shrinkage and that gradient boosting does not need to compute the weights W k , since linear models are only fitted on the negative gradient −∂ℓ pen (β)/∂η θ k .…”

Section: Estimationmentioning

confidence: 99%

LASSO-type penalization in the framework of generalized additive models for location, scale and shape

Groll

Hambuckers

Kneib

et al. 2019

Computational Statistics & Data Analysis

View full text Add to dashboard Cite

For numerous applications, it is of interest to provide full probabilistic forecasts, which are able to assign plausibilities to each predicted outcome. Therefore, attention is shifting constantly from conditional mean models to probabilistic distributional models capturing location, scale, shape and other aspects of the response distribution. One of the most established models for distributional regression is the generalized additive model for location, scale and shape (GAMLSS). In high-dimensional data setups , classical fitting procedures for GAMLSS often become rather unstable and methods for variable selection are desirable. Therefore, a regularization approach for high-dimensional data setups in the framework of GAMLSS is proposed. It is designed for linear covariate effects and is based on L 1-type penalties. The following three penalization options are provided: the conventional least absolute shrinkage and selection operator (LASSO) for metric covariates, and both group and fused LASSO for categorical predictors. The methods are investigated both for simulated data and for two real data examples, namely Munich rent data and data on extreme operational losses from the Italian bank UniCredit.

show abstract

“…More importantly, the threshold can be chosen in such a way that the expected number of false discoveries can be theoretically bounded under mild conditions. Due to its flexibility and versatility, StabSel has received increasing popularity and been successfully applied in many domains since its inception.…”

Section: Related Workmentioning

confidence: 99%

“…A prominent feature of StabSel lies in its ability to control the FDR of the base learner through executing it multiple times on subsamples. Some scholars have argued that q Λ should be set relatively high, allowing for a certain number of false discoveries while increasing the chances that all truly important variables can be selected. Thus, we chose a false discovery tolerance of E ( V ) ≤ 4.…”

Section: Ensemble Pruning Of Stability Selectionmentioning

confidence: 99%

Pruning variable selection ensembles

Zhang

Zhu

2019

Statistical Analysis

View full text Add to dashboard Cite

In the context of variable selection, ensemble learning has gained increasing interest due to its great potential to improve selection accuracy and to reduce false discovery rate. A novel ordering-based selective ensemble learning strategy is designed in this paper to obtain smaller but more accurate ensembles. In particular, a greedy sorting strategy is proposed to rearrange the order by which the members are included into the integration process. Through stopping the fusion process early, a smaller subensemble with higher selection accuracy can be obtained. More importantly, the sequential inclusion criterion reveals the fundamental strength-diversity trade-off among ensemble members. By taking stability selection with the base learner lasso (abbreviated as SSLasso) as an example, some experiments are conducted to examine the performance of the novel algorithm. Experimental results demonstrate that pruned SSLasso generally achieves higher selection accuracy and lower false discovery rate than SSLasso and several other benchmark methods.

show abstract

Gradient boosting for distributional regression: faster tuning and improved variable selection via noncyclical updates

Cited by 67 publications

References 29 publications

Modeling spatiotemporal abundance of mobile wildlife in highly variable environments using boosted GAMLSS hurdle models

Modeling spatiotemporal abundance of mobile wildlife in highly variable environments using boosted GAMLSS hurdle models

LASSO-type penalization in the framework of generalized additive models for location, scale and shape

Pruning variable selection ensembles

Contact Info

Product

Resources

About