An iterative model-free feature screening procedure: Forward recursive selection

Xia, Siwei; Yang, Yuehan

doi:10.1016/j.knosys.2022.108745

Cited by 9 publications

(7 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, recursive feature elimination [58] is computationally demanding, as the iteration starts with all variables and ends when no variables remain. Forward recursive selection [61] is suitable for high-dimensional data, but the number of iterations is determined by the number of samples, leading to high computational costs when dealing with large datasets.…”

Section: Discussion and Comparison With Other Methods Andmentioning

confidence: 99%

“…A state-of-the-art approach to wrapper methods without model restrictions is recursive feature elimination (RFE), a sequential backward elimination, i.e., support vector machine-based recursive feature elimination [55][56][57], random forest-based recursive feature elimination [58,59], partial least squares-based recursive feature elimination [60]. Motivated by RFE, Xia and Yang [61] proposed an iterative model-free feature screening procedure named forward recursive selection.…”

Section: Motivation and Contributionmentioning

confidence: 99%

“…For example, Borboudakis and Tsamardinos [48] introduced early dropping to increase computational efciency. Honda and Lin [50] and Xia and Yang [61] reduced computational consumption by adding a stopping rule that takes into account the model size.…”

Section: Motivation and Contributionmentioning

confidence: 99%

See 2 more Smart Citations

A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination

Xia,

Yang

2023

International Journal of Intelligent Systems

Self Cite

View full text Add to dashboard Cite

This paper studies data with mass features, commonly observed in applications such as text classification and medical diagnosis. We allow data to have several structures without requiring a specific model and propose an efficient model-free feature selection procedure. The proposed method can work with various types of datasets. We demonstrate that this method has several desirable properties, including high accuracy, model-free, and computational efficiency and can be applied to practical problems with different modelings. We prove that the proposed method achieves selection consistency and L 2 consistency under mild regularity conditions. We conduct simulations on various datasets, including data generated from the generalized linear model, additive model, Poisson regression, and binary classification model. These simulations illustrate the superior performance of the proposed method compared to other existing methods across different model settings. In addition, we apply our method to two real examples, the Tecator dataset and the Daily Demand Orders dataset, both of which are continuous and high dimensional. In both cases, our method consistently achieves high accuracy in prediction and model selection.

show abstract

Section: Discussion and Comparison With Other Methods Andmentioning

confidence: 99%

Section: Motivation and Contributionmentioning

confidence: 99%

See 1 more Smart Citation

A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination

Xia,

Yang

2023

International Journal of Intelligent Systems

Self Cite

View full text Add to dashboard Cite

show abstract

“…Many techniques have been suggested on how to discard non-significant variables. 4,17,25,27 The most successful algorithms for this purpose are the wrapper methods, 24,[28][29][30][31][32][33] which return a final subset of all-relevant variables. To efficiently use a wrapper method, the model to be used should be both computationally efficient and simple, with no user-defined parameters if possible, which is the case for the RF models.…”

Section: Introductionmentioning

confidence: 99%

Variable selection wrapper in presence of correlated input variables for random forest models

Rotari,

Kulahci

2023

Quality & Reliability Eng

View full text Add to dashboard Cite

In most data analytic applications in manufacturing, understanding the data‐driven models plays a crucial role in complementing the engineering knowledge about the production process. Identifying relevant input variables, rather than only predicting the response through some “black‐box” model, is of great interest in many applications. There is, therefore, a growing focus on describing the contributions of the input variables to the model in the form of “variable importance”, which is readily available in certain machine learning methods such as random forest (RF). Once a ranking based on the importance measure of the variables is established, the question of how many variables are truly relevant in predicting the output variable rises. In this study, we focus on the Boruta algorithm, which is a wrapper around the RF model. It is a variable selection tool that assesses the variable importance measure for the RF model. It has been previously shown in the literature that the correlation among the input variables, which is often a common occurrence in high dimensional data, distorts and overestimates the importance of variables. The Boruta algorithm is also affected by this resulting in a larger set of input variables deemed important. To overcome this issue, in this study, we propose an extension of the Boruta algorithm for the correlated data by exploiting the conditional importance measure. This extension greatly improves the Boruta algorithm in the case of high correlation among variables and provides a more precise ranking of the variables that significantly contribute to the response. We believe this approach can be used in many industrial applications by providing more transparency and understanding of the process.

show abstract

“…It first selects coefficient block according to the canonical correlation coefficients and then selects nonzero coefficients in blocks by the EBIC (Chen and Chen, 2008) and often surpasses the MSGL in both accuracy and computation. Further, Xia and Yang (2022) has studied a stepwise algorithm with the permutation importance measure, and Luo and Chen (2020) has discussed the principle of correlations. Sequential procedures also have been wild studied in other fields, such as decision making (Liu et al, 2020), fMRI data analysis (Fan et al, 2020), multidimensional time series (Li et al, 2021), structural changes in time series (Kejriwal, 2020), parameters estimation for radio equipment (Zaliskyi et al, 2021), etc.…”

Section: Introductionmentioning

confidence: 99%

A sequential stepwise screening procedure for sparse recovery in high-dimensional multiresponse models with complex group structures

Wei-xiong¹,

Yang²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Multiresponse data with complex group structures in both responses and predictors arises in many fields, yet, due to the difficulty in identifying complex group structures, only a few methods have been studied on this problem. We propose a novel algorithm called sequential stepwise screening procedure (SeSS) for feature selection in high-dimensional multiresponse models with complex group structures. This algorithm encourages the grouping effect, where responses and predictors come from different groups, further, each response group is allowed to relate to multiple predictor groups. To obtain a correct model under the complex group structures, the proposed procedure first chooses the nonzero block and the nonzero row by the canonical correlation measure (CC) and then selects the nonzero entries by the extended Bayesian Information Criterion (EBIC). We show that this method is accurate in extremely sparse models and computationally attractive. The theoretical property of SeSS is established. We conduct simulation studies and consider a real example to compare its performances with existing methods.

show abstract

An iterative model-free feature screening procedure: Forward recursive selection

Cited by 9 publications

References 34 publications

A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination

A Model-Free Feature Selection Technique of Feature Screening and Random Forest-Based Recursive Feature Elimination

Variable selection wrapper in presence of correlated input variables for random forest models

A sequential stepwise screening procedure for sparse recovery in high-dimensional multiresponse models with complex group structures

Contact Info

Product

Resources

About