Model selection for inferential models with high dimensional data: synthesis and graphical representation of multiple techniques

Abstract

Inferential research commonly involves identification of causal factors from within high dimensional data but selection of the ‘correct’ variables can be problematic. One specific problem is that results vary depending on statistical method employed and it has been argued that triangulation of multiple methods is advantageous to safely identify the correct, important variables. To date, no formal method of triangulation has been reported that incorporates both model stability and coefficient estimates; in this paper we develop an adaptable, straightforward method to achieve this. Six methods of variable selection were evaluated using simulated datasets of different dimensions with known underlying relationships. We used a bootstrap methodology to combine stability matrices across methods and estimate aggregated coefficient distributions. Novel graphical approaches provided a transparent route to visualise and compare results between methods. The proposed aggregated method provides a flexible route to formally triangulate results across any chosen number of variable selection methods and provides a combined result that incorporates uncertainty arising from between-method variability. In these simulated datasets, the combined method generally performed as well or better than the individual methods, with low error rates and clearer demarcation of the true causal variables than for the individual methods.

Selection Stability in High Dimensional Statistical Modelling: Defining a Threshold for Robust Model Inference

“…In terms of the substantial improvements in accuracy of covariate selection, the additional time required would appear to be very worthwhile. Also worthy of note in our results was the excellent performance of the combination modelling method; this is in agreement with previous research 19 . For all datasets this method provided the most accurate variable selection and this is likely to be due to the principle of triangulation.…”

“…For each method, a conventional approach to covariate selection was conducted followed by implementation of selection stability by bootstrapping 14 . The modelling methods chosen are acknowledged approaches for inferential modelling and were stepwise selection based on Akaike information criterion (sAIC) 17 , stepwise selection based on a modified Bayesian Information Criterion (mBIC) 5 , elastic net regression (enet) 8 , minimax convex penalty regression (MCP) 18 and a combination method that synthesised results from all four methods 19 . These modelling techniques are described in detail in Section 2.1 and the new approach to stability selection, including the determination of a threshold for inference, is described in Section 2.2.…”

“…Covariate selection stabilities were combined across all modelling methods using an 170 approach described previously 19 . In brief, the bootstrap matrices from each individual modelling method were aligned by covariate to create a single matrix containing an equal 172 number of bootstrap samples from each method.…”

“…Graphical analysis was used to visualise model outcomes; scatterplots were used to illustrate results of the stability analysis by plotting the stability (%) against BPV for covariates selected in the model, as previously reported 19 but with the additional annotation of the selection stability threshold.…”

Cow- and herd-level risk factors for lameness in partly housed pasture-based dairy cows

“…Triangulation (Lawlor et al, 2017;Lima et al, 2021) of results from elastic net regression (Enet), a form of regularized logistic regression, and logistic regression using modified Bayesian information criterion (mBIC) was used to establish important risk factors for lameness. These methods were chosen due to the large number of predictors and the need to avoid overfitting.…”