Predictive Abilities of Machine Learning Techniques May Be Limited by Dataset Characteristics: Insights From the UNOS Database

Miller, Paul E.; Pawar, Sumeet; Vaccaro, Benjamin J.; McCullough, Megan; Rao, Pooja; Ghosh, Rohit; Warier, Prashant; Desai, Nihar R.; Ahmad, Tariq

doi:10.1016/j.cardfail.2019.01.018

Cited by 53 publications

(54 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

Section: Discussionmentioning

confidence: 72%

“…Prior studies have assessed the performance of clinical predictive models, finding, as in our study, that machine learning methods performed equivalently to standard regression analyses. [27][28][29] Although advanced analytic methods and traditional regression models have comparable discrimination, model performance is often influenced by both the size of the cohort under study and the number of events per variable (EPV). [30][31][32][33] Evidence suggests that logistic regression models perform better (in terms of accuracy, parsimony and/or discrimination) in smaller datasets with approximately 20-50 EPV, while random forest models perform well with larger sample sizes and achieve sufficient stability when EPV exceeds 200.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Novel application of approaches to predicting medication adherence using medical claims data

Zullig

Jazowski

Wang

et al. 2019

Health Services Research

View full text Add to dashboard Cite

Objective To compare predictive analytic approaches to characterize medication nonadherence and determine under which circumstances each method may be best applied. Data Sources/Study Setting Medicare Parts A, B, and D claims from 2007 to 2013. Study Design We evaluated three statistical techniques to predict statin adherence (proportion of days covered [PDC ≥ 80 percent]) in the year following discharge: standard logistic regression with backward selection of covariates, least absolute shrinkage and selection operator (LASSO), and random forest. We used the C‐index to assess model discrimination and decile plots comparing predicted values to observed event rates to evaluate model performance. Data Extraction We identified 11 969 beneficiaries with an acute myocardial infarction (MI)‐related admission from 2007 to 2012, who filled a statin prescription at, or shortly after, discharge. Principal Findings In all models, prior statin use was the most important predictor of future adherence (OR = 3.65, 95% CI: 3.34‐3.98; OR = 3.55). Although the LASSO regression model selected nearly 90 percent of all candidate predictors, all three analytic approaches had moderate discrimination (C‐index ranging from 0.664 to 0.673). Conclusions Although none of the models emerged as clearly superior, predictive analytics could proactively determine which patients are at risk of nonadherence, thus allowing for timely engagement in adherence‐improving interventions.

show abstract

Section: Discussionmentioning

confidence: 72%

Section: Discussionmentioning

confidence: 99%

Novel application of approaches to predicting medication adherence using medical claims data

Zullig

Jazowski

Wang

et al. 2019

Health Services Research

View full text Add to dashboard Cite

show abstract

“…First, large multicenter registries, like the UNOS dataset, were the main cohort source for the derivate models. The lack of granularity of data included in these registries may be the most important limitation 35 . The UNOS Thoracic Committee recently decided to expand the collection of data to capture more prognostic markers in order to improve risk stratification.…”

Section: Discussionmentioning

confidence: 99%

“…The analysis of complex interactions between predictive variables may improve risk stratification (eg, donor age and ischemic time) 36‐38 . However, different machine learning approaches failed to improve the discrimination ability of predictive models 35 . We believe that international collaborations, at the level of centers, to build prospective prediction models based on a deep phenotyped database may increase the granularity of the dataset, heterogeneity of allocation schemes and practices, and finally, the statistical performance of these models.…”

Section: Discussionmentioning

confidence: 99%

Statistical performance of 16 posttransplant risk scores in a contemporary cohort of heart transplant recipients

Coutance

Kransdorf

Bonnet

et al. 2021

American Journal of Transplantation

View full text Add to dashboard Cite

Accurate risk stratification of early heart transplant failure is required to avoid futile transplants and rationalize donor selection. We aimed to evaluate the statistical performance of existing risk scores on a contemporary cohort of heart transplant recipients. After an exhaustive search, we identified 16 relevant risk scores. From the UNOS database, we selected all first noncombined adult heart transplants performed between 2014 and 2017 for validation. The primary endpoint was death or retransplant during the first year posttransplant. For all scores, we analyzed their association with outcomes, sensitivity, specificity, likelihood ratios, and discrimination (concordance index and overlap of individual scores). The cohort included 9396 patients. All scores were significantly associated with the primary outcome (P < .001 for all scores). Their likelihood ratios, both negative and positive, were poor. The discriminative performance of all scores was limited, with concordance index ranging from 0.544 to 0.646 (median 0.594) and an important overlap of individual scores between patients with or without the primary endpoint. Subgroup analyses revealed important variation in discrimination according to donor age, recipient age, and the type of assist device used at transplant. Our findings raise concerns about the use of currently available scores in the clinical field.

show abstract

“…Our findings are consistent with a study in heart transplantation by Miller et al . [34] that found no meaningful difference in predicting 1‐year survival between logistic regression and ML algorithms using the same set of variables, with C ‐statistics around 0.65 in most methods. We have extended this approach to kidney transplantation, to outcomes beyond 1 year, to Cox regression which is the typical method for evaluating survival, and to nonsurvival outcomes such as DGF and AR.…”

Section: Discussionmentioning

confidence: 99%

Machine learning to predict transplant outcomes: helpful or hype? A national cohort study

et al. 2020

View full text Add to dashboard Cite

An increasing number of studies claim machine learning (ML) predicts transplant outcomes more accurately. However, these claims were possibly confounded by other factors, namely, supplying new variables to ML models. To better understand the prospects of ML in transplantation, we compared ML to conventional regression in a "common" analytic task: predicting kidney transplant outcomes using national registry data. We studied 133 431 adult deceased-donor kidney transplant recipients between 2005 and 2017. Transplant centers were randomly divided into 70% training set (190 centers/97 787 recipients) and 30% validation set (82 centers/ 35 644 recipients). Using the training set, we performed regression and ML procedures [gradient boosting (GB) and random forests (RF)] to predict delayed graft function, one-year acute rejection, death-censored graft failure C, all-cause graft failure, and death. Their performances were compared on the validation set using-statistics. In predicting rejection, regression (C = 0.601 0.611 0.621) actually outperformed GB (C = 0.581 0.591 0.601) and RF (C = 0.569 0.579 0.589). For all other outcomes, the C-statistics were nearly identical across methods (delayed graft function, 0.717-0.723; death-censored graft failure, 0.637-0.642; all-cause graft failure, 0.633-0.635; and death, 0.705-0.708). Given its shortcomings in model interpretability and hypothesis testing, ML is advantageous only when it clearly outperforms conventional regression; in the case of transplant outcomes prediction, ML seems more hype than helpful.

show abstract

Predictive Abilities of Machine Learning Techniques May Be Limited by Dataset Characteristics: Insights From the UNOS Database

Cited by 53 publications

References 17 publications

Novel application of approaches to predicting medication adherence using medical claims data

Novel application of approaches to predicting medication adherence using medical claims data

Statistical performance of 16 posttransplant risk scores in a contemporary cohort of heart transplant recipients

Machine learning to predict transplant outcomes: helpful or hype? A national cohort study

Contact Info

Product

Resources

About