Variable selection for inferential models with relatively high-dimensional data: Between method heterogeneity and covariate stability as adjuncts to robust selection

Abstract

Variable selection in inferential modelling is problematic when the number of variables is large relative to the number of data points, especially when multicollinearity is present. A variety of techniques have been described to identify 'important' subsets of variables from within a large parameter space but these may produce different results which creates difficulties with inference and reproducibility. our aim was evaluate the extent to which variable selection would change depending on statistical approach and whether triangulation across methods could enhance data interpretation. A real dataset containing 408 subjects, 337 explanatory variables and a normally distributed outcome was used. We show that with model hyperparameters optimised to minimise cross validation error, ten methods of automated variable selection produced markedly different results; different variables were selected and model sparsity varied greatly. Comparison between multiple methods provided valuable additional insights. Two variables that were consistently selected and stable across all methods accounted for the majority of the explainable variability; these were the most plausible important candidate variables. Further variables of importance were identified from evaluating selection stability across all methods. In conclusion, triangulation of results across methods, including use of covariate stability, can greatly enhance data interpretation and confidence in variable selection.

Model selection for inferential models with high dimensional data: synthesis and graphical representation of multiple techniques

“…Importantly, our results also confirm the recently highlighted issue that different analytic methods used on same data can yield different results 11 , both in terms of variables selected and coefficient estimates 9 . The simulated datasets used in this study, in which the true underlying relationships were known, were useful to illustrate such differences between methods.…”

“…Over recent years, methods have been proposed in the statistical literature to improve variable selection for inference in high dimensional data, including modifications to AIC/BIC 5 , and a variety of regularisation methods based on functions that penalise model coefficients to balance over-and under-fitting (the variance-bias trade off) [6][7][8] . It has been shown, however, that different methods of variable selection can result in considerable differences in covariates selected 9 and this poses difficult questions for the researcher about which method to choose, as well as presenting wider concerns around variability of results and therefore the reproducibility of science 10,11 .…”

“…Triangulation of multiple methods has been proposed as an aid to identify important variables 13 ; in this context triangulation refers to conducting a variety of analytic methods on one set of data, on the premise that the most important variables will tend to be identified by most methods. Indeed, recent research has indicated this approach is likely to be beneficial 9 . However, rather than using triangulation to simply compare methods, a route to formally combine results from several statistical approaches would be advantageous to explicitly represent the additional uncertainty arising from variation between methods.…”

“…used to estimate covariate stability for all analytic approaches, according to methods previously described 9 . In brief, selection stability 14,15,20 was evaluated for each model as the percentage of times that each covariate was selected in the model across bootstrap samples.…”

Reanalysis datasets outperform other gridded climate products in vegetation change analysis in peripheral conservation areas of Central Asia

“…As results are expected to be variable due to the high dimensionality and the comparably low number of available years, we additionally assessed the stability of suitable datasets using different variable selection approaches in a prediction context. We applied two different methods in this study, as a comparison of different approaches is suggested if modeling is performed with high dimensional data 89 . One approach, which is methodologically comparable to our correlation analysis, is the calculation of models with single stepwise forward selection based on Pearson’s correlation and 100 repeated, threefold cross-validation cf.…”