Knot selection by boosting techniques

Abstract

A novel concept for estimating smooth functions by selection techniques based on boosting is developed. It is suggested to put radial basis functions with different spreads at each knot and to do selection and estimation simultaneously by a componentwise boosting algorithm. The methodology of various other smoothing and knot selection procedures (e.g. stepwise selection) is summarized. They are compared to the proposed approach by extensive simulations for various unidimensional settings, including varying spatial variation and heteroskedasticity, as well as on a real world data example. Finally, an extension of the proposed method to surface fitting is evaluated numerically on both, simulation and real data. The proposed knot selection technique is shown to be a strong competitor to existing methods for knot selection.

Boosting Algorithms: Regularization, Prediction and Model Fitting

“…Models for multivariate responses are studied in [20,59]; some multiclass boosting methods are discussed in [33,95]. Other works deal with boosting approaches for generalized linear and nonparametric models [55,56,85,86], for flexible semiparametric mixed models [88] or for nonparametric models with quality constraints [54,87]. Boosting methods for estimating propensity scores, a special weighting scheme for modeling observational data, are proposed in [63].…”

Semiparametric regression during 2003–2007

“…Versions of AIC and BIC are used as stopping criteria. Leitenstorfer & Tutz [160] use boosting for knot selection in a regression spline approach to smoothing.…”

“…Leitenstorfer & Tutz [160] also achieve spatial adaptive via model selection on the knots and a version of boosting.…”

Boosting additive models using component-wise P-Splines

“…As an example, Ruppert et al (2003) used a truncated power series basis for their penalized spline approach. In a boosting context, Bühlmann (2006) used thin plate splines, while Leitenstorfer and Tutz (2007) used a base-learner constructed from radial basis functions. Future work should thus include an investigation on the effect of different spline bases on the performance of L 2 Boosting.…”