Improving Practices for Selecting a Subset of Important Predictors in Psychology: An Application to Predicting Pain

Abstract

Frequently, researchers in psychology are faced with the challenge of narrowing down a large set of predictors to a smaller subset. There are a variety of ways to do this, but commonly it is done by choosing predictors with the strongest bivariate correlations with the outcome. However, when predictors are correlated, bivariate relationships may not translate into multivariate relationships. Further, any attempts to control for multiple testing are likely to result in extremely low power. Here we introduce a Bayesian variable-selection procedure frequently used in other disciplines, stochastic search variable selection (SSVS). We apply this technique to choosing the best set of predictors of the perceived unpleasantness of an experimental pain stimulus from among a large group of sociocultural, psychological, and neurobiological (functional MRI) individual-difference measures. Using SSVS provides information about which variables predict the outcome, controlling for uncertainty in the other variables of the model. This approach yields new, useful information to guide the choice of relevant predictors. We have provided Web-based open-source software for performing SSVS and visualizing the results.

The ideological basis of antiscientific attitudes: Effects of authoritarianism, conservatism, religiosity, social dominance, and system justification

“…As a means of avoiding overinflation and significant fluctuation in coefficients of correlated variables, we conducted a series of regularized regressions to identify a subset of predictors that exhibited the most robust effects via elastic nets (Zou & Hastie, 2005). We cross-validated these analyses using several other statistical techniques (backward, forward, and stepwise regression as well as k-fold cross-validation) to ensure that the results were not produced by idiosyncratic methodological choices (Bainter et al, 2020;de Rooij & Weeda, 2020).…”

Modality-specific associations of sensory deficits with autistic traits

“…We tested for normal distributions using the Shapiro-Wilk tests. To explore which sensory modalities may be associated with higher QAT, we used SSVS (Bainter et al, 2020). SSVS is a Bayesian framework used for empirically driven variable selection (George & McCulloch, 1993).…”

“…The SVSS uses Markov chain Monte Carlo sampling to sample from a posterior distribution of the possible subsets of predictors to identify the best models. Predictors selected more frequently in the sampling receive higher marginal inclusion probabilities (MIPs) (0.0-1.0) (Bainter et al, 2020;George & McCulloch, 1993). This approach selects predictors while controlling for uncertainty in other predictors included in the model, maximizing power, and minimizing false positives.…”

“…For each cohort, three analyses were performed using an online application that performs SSVS (Bainter et al, 2020; https://ssvsforpsych.shinyapps.io/ssvsforpsych/). All analyses used the following SSVS specifications: prior inclusion probability α = 0.5 (indicating that each predictor has a 50/50 prior probability of being included in the model), 5,000 burn-in iterations to achieve convergence, and 20,000 total iterations.…”

“…Comparisons of genetic, biological, and behavioral contributions of specific sensory modalities to autistic traits are complicated by high collinearity between constructs (Hurley et al, 2007; One way of assessing the relative importance of intercorrelated predictor variables is dominance analysis (DA), which can provide information about the contribution of individual predictors to the total R 2 of a multiple regression model (Braun et al, 2019). To evaluate the role of different modalities in predicting specific autistic traits, we combined DA with stochastic search variable selection (SSVS; Bainter et al, 2020). We recruited two independent cohorts of adults to complete the GSQ and BAPQ (N = 252 + 268).…”