A Novel Metric for Developing Easy-to-Use and Accurate Clinical Prediction Models

Lee, Sei J.; Smith, Alexander K.; Diaz‐Ramirez, L. Grisell; Covinsky, Kenneth E.; Gan, Siqi; Chen, Catherine L.; Boscardin, W. John

doi:10.1097/mlr.0000000000001510

Cited by 5 publications

(9 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Statistically, this state-of-the-art prognostic tool used three recently developed methods that enabled variable selection for three outcomes simultaneously, while also considering the time-cost of each variable in addition to its predictive power. 10,11,18 Our models had good discrimination and calibration with little evidence of overfitting, and the models performed well even when one or two predictors were unavailable. This prognostic tool fills an important gap in the literature to predict patient-centered outcomes of ADL disability and walking disability for older adults in the US.…”

Section: Discussionmentioning

confidence: 84%

“…We applied statistical methods we previously developed to conduct backward elimination on three prediction models simultaneously and accounting for the time needed to assess a predictor using the time-cost information criterion (TCIC). 10,18 Briefly, the TCIC is an alternative to the BIC used to evaluate model fit, and the TCIC favors the variable that takes less time to assess when two variables have identical predictive abilities. Backward elimination was conducted on all three models simultaneously by applying the TCIC to each model at each backward elimination step and finding the lowest average TCIC across the three models.…”

Section: Discussionmentioning

confidence: 99%

“…To determine the time cost for each predictor, we used the automated reading function of Microsoft Excel. We measured the time it took Excel's automated reader to read verbatim each question from HRS (as listed in Table S1) and used the number of seconds as the time–cost for each predictor 10 …”

Section: Methodsmentioning

confidence: 99%

“…However, given the highly limited clinical encounter time, it is crucial to not further burden healthcare providers by having different tools for each outcome. Thus, it is important to build prediction models using an approach that considers the clinical time needed to assess that variable in addition to its predictive capacity 10 . Additionally, patient data is sometimes unavailable, so it is important to enable predictions even with one or two missing predictors 11 …”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A comprehensive prognostic tool for older adults: Predicting death, ADL disability, and walking disability simultaneously

Lee¹,

Diaz‐Ramirez²,

Boscardin³

et al. 2022

J American Geriatrics Society

Self Cite

View full text Add to dashboard Cite

Background Many clinical and financial decisions for older adults depend on the future risk of disability and mortality. Prognostic tools for long‐term disability risk in a general population are lacking. We aimed to create a comprehensive prognostic tool that predicts the risk of mortality, of activities of daily living (ADL) disability, and walking disability simultaneously using the same set of variables. Methods We conducted a longitudinal analysis of the nationally‐representative Health and Retirement Study (HRS). We included community‐dwelling adults aged ≥70 years who completed a core interview in the 2000 wave of HRS, with follow‐up through 2018. We evaluated 40 predictors encompassing demographics, diseases, physical functioning, and instrumental ADLs. We applied novel methods to optimize three models simultaneously while prioritizing variables that take less time to ascertain during backward stepwise elimination. The death prediction model used Cox regression and both the models for walking disability and for ADL disability used Fine and Gray competing‐risk regression. We examined calibration plots and generated optimism‐corrected statistics of discrimination using bootstrapping. To simulate unavailable patient data, we also evaluated models excluding one or two variables from the final model. Results In 6646 HRS participants, 2662 developed walking disability, 3570 developed ADL disability, and 5689 died during a median follow‐up of 9.5 years. The final prognostic tool had 16 variables. The optimism‐corrected integrated area under the curve (iAUC) was 0.799 for mortality, 0.685 for walking disability, and 0.703 for ADL disability. At each percentile of predicted mortality risk, there was a substantial spread in the predicted risks of walking disability and ADL disability. Discrimination and calibration remained good even when missing one or two predictors from the model. This model is now available on ePrognosis (https://eprognosis.ucsf.edu/alexlee.php) Conclusions Given the variability in disability risk for people with similar mortality risks, using individualized risks of disabilities may inform clinical and financial decisions for older adults.

show abstract

Section: Discussionmentioning

confidence: 84%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

A comprehensive prognostic tool for older adults: Predicting death, ADL disability, and walking disability simultaneously

Lee¹,

Diaz‐Ramirez²,

Boscardin³

et al. 2022

J American Geriatrics Society

Self Cite

View full text Add to dashboard Cite

show abstract

“…From a practical perspective, busy clinicians are unlikely to use a prognostic model with a daunting list of predictors to collect and enter. Although we did not formally incorporate a penalization associated with the cost of the predictors, other authors have explicitly balanced predictive accuracy against cost of the predictors [14,33].…”

Section: Discussionmentioning

confidence: 99%

A Novel Method for Identifying a Parsimonious and Accurate Predictive Model for Multiple Clinical Outcomes

Diaz‐Ramirez

Lee

Smith

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Background and Objective: Most methods for developing clinical prognostic models focus on identifying parsimonious and accurate models to predict a single outcome; however, patients and providers often want to predict multiple outcomes simultaneously. As an example, for older adults one is often interested in predicting nursing home admission as well as mortality. We propose and evaluate a novel predictor-selection computing method for multiple outcomes and provide the code for its implementation.Methods: Our proposed algorithm selected the best subset of common predictors based on the minimum average normalized Bayesian Information Criterion (BIC) across outcomes: the Best Average BIC (baBIC) method. We compared the predictive accuracy (Harrell’s C-statistic) and parsimony (number of predictors) of the model obtained using the baBIC method with: 1) a subset of common predictors obtained from the union of optimal models for each outcome (Union method), 2) a subset obtained from the intersection of optimal models for each outcome (Intersection method), and 3) a model with no variable selection (Full method). We used a case-study data from the Health and Retirement Study (HRS) to demonstrate our method and conducted a simulation study to investigate performance.Results: In the case-study data and simulations, the average Harrell’s C-statistics across outcomes of the models obtained with the baBIC and Union methods were comparable. Despite the similar discrimination, the baBIC method produced more parsimonious models than the Union method. In contrast, the models selected with the Intersection method were the most parsimonious, but with worst predictive accuracy, and the opposite was true in the Full method. In the simulations, the baBIC method performed well by identifying many of the predictors selected in the baBIC model of the case-study data most of the time and excluding those not selected in the majority of the simulations.Conclusions: Our method identified a common subset of variables to predict multiple clinical outcomes with superior balance between parsimony and predictive accuracy to current methods.

show abstract

A Novel Method for Identifying a Parsimonious and Accurate Predictive Model for Multiple Clinical Outcomes

Diaz‐Ramirez

Lee

Smith

et al. 2021

Computer Methods and Programs in Biomedicine

Self Cite

View full text Add to dashboard Cite

Background: Most methods for developing clinical prognostic models focus on identifying parsimonious and accurate models to predict a single outcome; however, patients and providers often want to predict multiple outcomes simultaneously. For example, older adults are often interested in predicting nursing home admission as well as mortality. We propose and evaluate a novel predictor selection method for multiple outcomes.Methods: Our proposed method selected the best subset of common predictors based on the minimum average normalized Bayesian Information Criterion (BIC) across outcomes: the Best Average BIC (baBIC) model. We compared the predictive accuracy (Harrell's C-statistic) and parsimony (number of predictors) of the baBIC model with a subset of common predictors obtained from the union of optimal models for each outcome (Union model). We used example data from the Health and Retirement Study (HRS) to demonstrate our method and conducted a simulation study to investigate performance considering correlated and uncorrelated outcomes. Results:In the example data, the average Harrell's C-statistics across outcomes of the baBIC and Union models were comparable (0.657 vs. 0.662 respectively). Despite the similar discrimination, the baBIC model was more parsimonious than the Union model (15 vs. 23 predictors respectively).Likewise, in the simulations with correlated outcomes, the mean C-statistic across outcomes of the baBIC and Union models were the same after rounding: 0.650, and the baBIC model had an average

show abstract

A Novel Metric for Developing Easy-to-Use and Accurate Clinical Prediction Models

Abstract: Supplemental Digital Content is available in the text.

Cited by 5 publications

References 27 publications

A comprehensive prognostic tool for older adults: Predicting death, ADL disability, and walking disability simultaneously

A comprehensive prognostic tool for older adults: Predicting death, ADL disability, and walking disability simultaneously

A Novel Method for Identifying a Parsimonious and Accurate Predictive Model for Multiple Clinical Outcomes

A Novel Method for Identifying a Parsimonious and Accurate Predictive Model for Multiple Clinical Outcomes

Contact Info

Product

Resources

About