Comparison of variable selection methods for clinical predictive modeling

Sanchez-Pinto, L. Nelson; Venable, Laura Ruth; Fahrenbach, John; Churpek, Matthew M.

doi:10.1016/j.ijmedinf.2018.05.006

Cited by 184 publications

(161 citation statements)

References 32 publications

(43 reference statements)

Supporting

Mentioning

158

Contrasting

Unclassified

Order By: Relevance

“…In model 2, from the 7 traditional and 11 additional factors, relevant predictors were selected using backward selection, with the AIC stopping rule . This is an extensively used selection method in clinical prediction that seems to achieve better parsimony in smaller datasets compared to modern tree‐based methods . Variables that were selected in >33% of the 20 imputed datasets were kept in the model.…”

Section: Methodsmentioning

confidence: 99%

“…23 This is an extensively used selection method in clinical prediction 23,24 that seems to achieve better parsimony in smaller datasets compared to modern tree-based methods. 23 Variables that were selected in >33% of the 20 imputed datasets were kept in the model. Age was forced in the model, since previous studies agreed on age as a risk factor, and the age range in this population was limited.…”

Section: Statistical Analysesmentioning

confidence: 99%

See 1 more Smart Citation

A cardiovascular risk prediction model for older people: Development and validation in a primary care population

Bussel

Richard

Busschers

et al. 2019

J of Clinical Hypertension

View full text Add to dashboard Cite

Cardiovascular risk prediction is mainly based on traditional risk factors that have been validated in middle‐aged populations. However, associations between these risk factors and cardiovascular disease (CVD) attenuate with increasing age. Therefore, for older people the authors developed and internally validated risk prediction models for fatal and non‐fatal CVD, (re)evaluated the predictive value of traditional and new factors, and assessed the impact of competing risks of non‐cardiovascular death. Post hoc analyses of 1811 persons aged 70‐78 year and free from CVD at baseline from the preDIVA study (Prevention of Dementia by Intensive Vascular care, 2006‐2015), a primary care‐based trial that included persons free from dementia and conditions likely to hinder successful long‐term follow‐up, were performed. In 2017‐2018, Cox‐regression analyses were performed for a model including seven traditional risk factors only, and a model to assess incremental predictive ability of the traditional and eleven new factors. Analyses were repeated accounting for competing risk of death, using Fine‐Gray models. During an average of 6.2 years of follow‐up, 277 CVD events occurred. Age, sex, smoking, and type 2 diabetes mellitus were traditional predictors for CVD, whereas total cholesterol, HDL‐cholesterol, and systolic blood pressure (SBP) were not. Of the eleven new factors, polypharmacy and apathy symptoms were predictors. Discrimination was moderate (concordance statistic 0.65). Accounting for competing risks resulted in slightly smaller predicted absolute risks. In conclusion, we found, SBP, HDL, and total cholesterol no longer predict CVD in older adults, whereas polypharmacy and apathy symptoms are two new relevant predictors. Building on the selected risk factors in this study may improve CVD prediction in older adults and facilitate targeting preventive interventions to those at high risk.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Statistical Analysesmentioning

confidence: 99%

A cardiovascular risk prediction model for older people: Development and validation in a primary care population

Bussel

Richard

Busschers

et al. 2019

J of Clinical Hypertension

View full text Add to dashboard Cite

show abstract

“…Prior studies have assessed the performance of clinical predictive models, finding, as in our study, that machine learning methods performed equivalently to standard regression analyses . Although advanced analytic methods and traditional regression models have comparable discrimination, model performance is often influenced by both the size of the cohort under study and the number of events per variable (EPV) . Evidence suggests that logistic regression models perform better (in terms of accuracy, parsimony and/or discrimination) in smaller datasets with approximately 20‐50 EPV, while random forest models perform well with larger sample sizes and achieve sufficient stability when EPV exceeds 200 .…”

Section: Discussionmentioning

confidence: 60%

“…In addition, the number and type of variables selected as predictors affect model performance. Previous research has demonstrated that random forest models achieve high performance not only as more variables are selected, but also when a large number of continuous variables are used as predictors . Given these considerations, data composition, quality, and completeness should be of the utmost importance when selecting or merging clinical data sources for prediction modeling.…”

Section: Discussionmentioning

confidence: 99%

“…[27][28][29] Although advanced analytic methods and traditional regression models have comparable discrimination, model performance is often influenced by both the size of the cohort under study and the number of events per variable (EPV). [30][31][32][33] Evidence suggests that logistic regression models perform better (in terms of accuracy, parsimony and/or discrimination) in smaller datasets with approximately 20-50 EPV, while random forest models perform well with larger sample sizes and achieve sufficient stability when EPV exceeds 200. 30,31,34,35 In addition, the number and type of variables selected as predictors affect model performance.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Novel application of approaches to predicting medication adherence using medical claims data

Zullig

Jazowski

Wang

et al. 2019

Health Services Research

View full text Add to dashboard Cite

Objective To compare predictive analytic approaches to characterize medication nonadherence and determine under which circumstances each method may be best applied. Data Sources/Study Setting Medicare Parts A, B, and D claims from 2007 to 2013. Study Design We evaluated three statistical techniques to predict statin adherence (proportion of days covered [PDC ≥ 80 percent]) in the year following discharge: standard logistic regression with backward selection of covariates, least absolute shrinkage and selection operator (LASSO), and random forest. We used the C‐index to assess model discrimination and decile plots comparing predicted values to observed event rates to evaluate model performance. Data Extraction We identified 11 969 beneficiaries with an acute myocardial infarction (MI)‐related admission from 2007 to 2012, who filled a statin prescription at, or shortly after, discharge. Principal Findings In all models, prior statin use was the most important predictor of future adherence (OR = 3.65, 95% CI: 3.34‐3.98; OR = 3.55). Although the LASSO regression model selected nearly 90 percent of all candidate predictors, all three analytic approaches had moderate discrimination (C‐index ranging from 0.664 to 0.673). Conclusions Although none of the models emerged as clearly superior, predictive analytics could proactively determine which patients are at risk of nonadherence, thus allowing for timely engagement in adherence‐improving interventions.

show abstract

Patient-Reported Social and Behavioral Determinants of Health and Estimated Risk of Hospitalization in High-Risk Veterans Affairs Patients

et al. 2020

View full text Add to dashboard Cite

IMPORTANCE Despite recognition of the association between individual social and behavioral determinants of health (SDH) and patient outcomes, little is known regarding the value of SDH in explaining variation in outcomes for high-risk patients. OBJECTIVE To describe SDH factors among veterans who are at high risk for hospitalization, and to determine whether adding patient-reported SDH measures to electronic health record (EHR) measures improves estimation of 90-day and 180-day all-cause hospital admission. DESIGN, SETTING, AND PARTICIPANTS A survey was mailed between April 16 and June 29, 2018, to a nationally representative sample of 10 000 Veterans Affairs (VA) patients whose 1-year risk of hospitalization or death was in the 75th percentile or higher based on a VA EHR-derived risk score. The survey included multiple SDH measures, such as resilience, social support, health literacy, smoking status, transportation barriers, and recent life stressors. MAIN OUTCOMES AND MEASURES The EHR-based characteristics of survey respondents and nonrespondents were compared using standardized differences. Estimation of 90-day and 180-day hospital admission risk was assessed for 3 logistic regression models: (1) a base model of all prespecified EHR-based covariates, (2) a restricted model of EHR-based covariates chosen via forward selection based on minimizing Akaike information criterion (AIC), and (3) a model of EHRand survey-based covariates chosen via forward selection based on AIC minimization. RESULTS In total, 4685 individuals (response rate 46.9%) responded to the survey. Respondents were comparable to nonrespondents in most characteristics, but survey respondents were older (eg, >80 years old, 881 [18.8%] vs 800 [15.1%]), comprised a higher percentage of men (4391 [93.7%] vs 4794 [90.2%]), and were composed of more White non-Hispanic individuals (3366 [71.8%] vs 3259 [61.3%]). Based on AIC, the regression model with survey-based covariates and EHR-based covariates better estimated hospital admission at 90 days (AIC, 1947.7) and 180 days (AIC, 2951.9) than restricted models with only EHR-based covariates (AIC, 1980.2 at 90 days; AIC, 2981.9 at 180 days). This result was due to inclusion of self-reported measures such as marital or partner status, health-related locus of control, resilience, smoking status, health literacy, and medication insecurity. CONCLUSIONS AND RELEVANCE Augmenting EHR data with patient-reported social information improved estimation of 90-day and 180-day hospitalization risk, highlighting specific SDH factors that might identify individuals who are at high risk for hospitalization.

show abstract

Comparison of variable selection methods for clinical predictive modeling

Cited by 184 publications

References 32 publications

A cardiovascular risk prediction model for older people: Development and validation in a primary care population

A cardiovascular risk prediction model for older people: Development and validation in a primary care population

Novel application of approaches to predicting medication adherence using medical claims data

Patient-Reported Social and Behavioral Determinants of Health and Estimated Risk of Hospitalization in High-Risk Veterans Affairs Patients

Contact Info

Product

Resources

About