Variable Selection via Regression Trees in the Presence of Irrelevant Variables

Chang, Youngjae

doi:10.1080/03610918.2012.674600

Cited by 4 publications

(8 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…25 Recursive random forests modeling is a commonly used strategy to overcome that problem. 47,48 In this work, one-fourth of the least important variables were removed from the next round of modeling.…”

Section: ■ Discussionmentioning

confidence: 99%

“…23 Based on variable importance of predictor, a series of recursive variable selection have been proposed in the literature. 24,25 Briefly, the variable importance in recursive RF for each predictor was first calculated. Then a portion of variables, say 20%, with the smallest importance will be eliminated, and a new model will be built using the remaining variables.…”

Section: ■ Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recursive Random Forests Enable Better Predictive Performance and Model Interpretation than Variable Selection by LASSO

Zhu¹,

Xin²,

2015

J. Chem. Inf. Model.

View full text Add to dashboard Cite

Variable selection is of crucial significance in QSAR modeling since it increases the model predictive ability and reduces noise. The selection of the right variables is far more complicated than the development of predictive models. In this study, eight continuous and categorical data sets were employed to explore the applicability of two distinct variable selection methods random forests (RF) and least absolute shrinkage and selection operator (LASSO). Variable selection was performed: (1) by using recursive random forests to rule out a quarter of the least important descriptors at each iteration and (2) by using LASSO modeling with 10-fold inner cross-validation to tune its penalty λ for each data set. Along with regular statistical parameters of model performance, we proposed the highest pairwise correlation rate, average pairwise Pearson's correlation coefficient, and Tanimoto coefficient to evaluate the optimal by RF and LASSO in an extensive way. Results showed that variable selection could allow a tremendous reduction of noisy descriptors (at most 96% with RF method in this study) and apparently enhance model's predictive performance as well. Furthermore, random forests showed property of gathering important predictors without restricting their pairwise correlation, which is contrary to LASSO. The mutual exclusion of highly correlated variables in LASSO modeling tends to skip important variables that are highly related to response endpoints and thus undermine the model's predictive performance. The optimal variables selected by RF share low similarity with those by LASSO (e.g., the Tanimoto coefficients were smaller than 0.20 in seven out of eight data sets). We found that the differences between RF and LASSO predictive performances mainly resulted from the variables selected by different strategies rather than the learning algorithms. Our study showed that the right selection of variables is more important than the learning algorithm for modeling. We hope that a standard procedure could be developed based on these proposed statistical metrics to select the truly important variables for model interpretation, as well as for further use to facilitate drug discovery and environmental toxicity assessment.

show abstract

Section: ■ Discussionmentioning

confidence: 99%

Section: ■ Introductionmentioning

confidence: 99%

Recursive Random Forests Enable Better Predictive Performance and Model Interpretation than Variable Selection by LASSO

Zhu¹,

Xin²,

2015

J. Chem. Inf. Model.

View full text Add to dashboard Cite

show abstract

“…Various learning algorithms are available for building prediction models from data. They demonstrate different sensitivities to irrelevant variables [14,[22][23][24][25]. Some learning algorithms attempt to directly reflect each variable's relevance or automatically discard irrelevant ones based on their intrinsic characteristics.…”

Section: Related Workmentioning

confidence: 99%

“…In the presence of many irrelevant variables, however, there is an increased likelihood of spurious node splits occurring on some irrelevant variables during the training [14]. A prediction model with a higher model complexity is more likely to contain many spurious node splits, which may degrade the prediction accuracy [22].…”

Section: Decision Tree (Dt)mentioning

confidence: 99%

Effect of Irrelevant Variables on Faulty Wafer Detection in Semiconductor Manufacturing

Kim

Kang

2019

Energies

View full text Add to dashboard Cite

Machine learning has been applied successfully for faulty wafer detection tasks in semiconductor manufacturing. For the tasks, prediction models are built with prior data to predict the quality of future wafers as a function of their precedent process parameters and measurements. In real-world problems, it is common for the data to have a portion of input variables that are irrelevant to the prediction of an output variable. The inclusion of many irrelevant variables negatively affects the performance of prediction models. Typically, prediction models learned by different learning algorithms exhibit different sensitivities with regard to irrelevant variables. Algorithms with low sensitivities are preferred as a first trial for building prediction models, whereas a variable selection procedure is necessarily considered for highly sensitive algorithms. In this study, we investigate the effect of irrelevant variables on three well-known representative learning algorithms that can be applied to both classification and regression tasks: artificial neural network, decision tree (DT), and k-nearest neighbors (k-NN). We analyze the characteristics of these learning algorithms in the presence of irrelevant variables with different model complexity settings. An empirical analysis is performed using real-world datasets collected from a semiconductor manufacturer to examine how the number of irrelevant variables affects the behavior of prediction models trained with different learning algorithms and model complexity settings. The results indicate that the prediction accuracy of k-NN is highly degraded, whereas DT demonstrates the highest robustness in the presence of many irrelevant variables. In addition, a higher model complexity of learning algorithms leads to a higher sensitivity to irrelevant variables.

show abstract

“…On the other hand, Young et al [27] and Toth and Eltinge [28] try to focus on the design of samples used in the training to infer the tree instead of the own learning algorithm. Nowadays, hybrids methods have turned into new tools to be explored such as the combination of feature selection methods with regression trees [29], the use of a regression tree to select the input data of a neural network [30] or ensemble of rules obtained by regression trees [31].…”

Section: Proposed Methodology: Regression Treesmentioning

confidence: 99%

Local models-based regression trees for very short-term wind speed prediction

et al. 2015

View full text Add to dashboard Cite

a b s t r a c tThis paper evaluates the performance of different types of Regression Trees (RTs) in a real problem of very short-term wind speed prediction from measuring data in wind farms. RT is a solidly established methodology that, contrary to other soft-computing approaches, has been under-explored in problems of wind speed prediction in wind farms. In this paper we comparatively evaluate eight different types of RTs algorithms, and we show that they are able obtain excellent results in real problems of very short-term wind speed prediction, improving existing classical and soft-computing approaches such as multi-linear regression approaches, different types of neural networks and support vector regression algorithms in this problem. We also show that RTs have a very small computation time, that allows the retraining of the algorithms whenever new wind speed data are collected from the measuring towers.

show abstract

Variable Selection via Regression Trees in the Presence of Irrelevant Variables

Cited by 4 publications

References 11 publications

Recursive Random Forests Enable Better Predictive Performance and Model Interpretation than Variable Selection by LASSO

Recursive Random Forests Enable Better Predictive Performance and Model Interpretation than Variable Selection by LASSO

Effect of Irrelevant Variables on Faulty Wafer Detection in Semiconductor Manufacturing

Local models-based regression trees for very short-term wind speed prediction

Contact Info

Product

Resources

About