Calibration drift in regression and machine learning models for acute kidney injury

Davis, Sharon E.; Lasko, Thomas A.; Chen, Guanhua; Siew, Edward D.; Matheny, Michael E.

doi:10.1093/jamia/ocx030

Cited by 203 publications

(185 citation statements)

References 64 publications

Supporting

Mentioning

177

Contrasting

Unclassified

Order By: Relevance

“…Although study sizes have increased over time (among studies published in 2017 and 2018, the median [IQR] sample size was 3464 [286-21,498]), studies based on fewer than 1000 patients have been regularly published in recent years. Six studies used data on more than 100,000 patients: one in 2015, one in 2017, and four in 2018 [2,[12][13][14][15][16].…”

Section: Type Of Machine Learningmentioning

confidence: 99%

Use of machine learning to analyse routinely collected intensive care unit data: a systematic review

et al. 2019

View full text Add to dashboard Cite

Background Intensive care units (ICUs) face financial, bed management, and staffing constraints. Detailed data covering all aspects of patients’ journeys into and through intensive care are now collected and stored in electronic health records: machine learning has been used to analyse such data in order to provide decision support to clinicians. Methods Systematic review of the applications of machine learning to routinely collected ICU data. Web of Science and MEDLINE databases were searched to identify candidate articles: those on image processing were excluded. The study aim, the type of machine learning used, the size of dataset analysed, whether and how the model was validated, and measures of predictive accuracy were extracted. Results Of 2450 papers identified, 258 fulfilled eligibility criteria. The most common study aims were predicting complications (77 papers [29.8% of studies]), predicting mortality (70 [27.1%]), improving prognostic models (43 [16.7%]), and classifying sub-populations (29 [11.2%]). Median sample size was 488 (IQR 108–4099): 41 studies analysed data on > 10,000 patients. Analyses focused on 169 (65.5%) papers that used machine learning to predict complications, mortality, length of stay, or improvement of health. Predictions were validated in 161 (95.2%) of these studies: the area under the ROC curve (AUC) was reported by 97 (60.2%) but only 10 (6.2%) validated predictions using independent data. The median AUC was 0.83 in studies of 1000–10,000 patients, rising to 0.94 in studies of > 100,000 patients. The most common machine learning methods were neural networks (72 studies [42.6%]), support vector machines (40 [23.7%]), and classification/decision trees (34 [20.1%]). Since 2015 (125 studies [48.4%]), the most common methods were support vector machines (37 studies [29.6%]) and random forests (29 [23.2%]). Conclusions The rate of publication of studies using machine learning to analyse routinely collected ICU data is increasing rapidly. The sample sizes used in many published studies are too small to exploit the potential of these methods. Methodological and reporting guidelines are needed, particularly with regard to the choice of method and validation of predictions, to increase confidence in reported findings and aid in translating findings towards routine use in clinical practice. Electronic supplementary material The online version of this article (10.1186/s13054-019-2564-9) contains supplementary material, which is available to authorized users.

show abstract

Section: Type Of Machine Learningmentioning

confidence: 99%

Use of machine learning to analyse routinely collected intensive care unit data: a systematic review

et al. 2019

View full text Add to dashboard Cite

show abstract

“…Demonstrating utility or validity on retrospective, in silico settings does not guarantee that the product will perform well in a different setting 57 or in a different time period. 86,87 The utility and validity of the product must be reassessed across time (Step 2b) and space (Step 2d). Evaluating a machine learning product on a hold-out and temporal validation set (Step 2a) is recommended before integrating a product into clinical care.…”

Section: Evaluate and Validatementioning

confidence: 99%

A Path for Translation of Machine Learning Products into Healthcare Delivery

Sendak¹,

D’Arcy²,

Kashyap³

et al. 2020

EMJ Innov

View full text Add to dashboard Cite

Despite enormous enthusiasm, machine learning models are rarely translated into clinical care and there is minimal evidence of clinical or economic impact. New conference venues and academic journals have emerged to promote the proliferating research; however, the translational path remains unclear. This review undertakes the first in-depth study to identify how machine learning models that ingest structured electronic health record data can be applied to clinical decision support tasks and translated into clinical practice. The authors complement their own work with the experience of 21 machine learning products that address problems across clinical domains and across geographic populations. Four phases of translation emerge: design and develop, evaluate and validate, diffuse and scale, and continuing monitoring and maintenance. The review highlights the varying approaches taken across each phase by teams building machine learning products and presents a discussion of challenges and opportunities. The translational path and associated findings are instructive to researchers and developers building machine learning products, policy makers regulating machine learning products, and health system leaders who are considering adopting a machine learning product.

show abstract

“…In other domains, like acute kidney injury prediction, for example, calibration has been shown to worsen over time. [50] Event rate shift alone was a primary reason for such drift, so practical frameworks incorporating these shifts in program evaluation are important. Our findings underline the cross-sectional impacts of event rate differences not only in calibration but in usefulness and expected utility analyses.…”

Section: Discussionmentioning

confidence: 99%

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

Walsh

Sharman

Hripcsak

2017

Journal of Biomedical Informatics

View full text Add to dashboard Cite

Background Prior to implementing predictive models in novel settings, analyses of calibration and clinical usefulness remain as important as discrimination, but they are not frequently discussed. Calibration is a model’s reflection of actual outcome prevalence in its predictions. Clinical usefulness refers to the utilities, costs, and harms of using a predictive model in practice. A decision analytic approach to calibrating and selecting an optimal intervention threshold may help maximize the impact of readmission risk and other preventive interventions. Objectives To select a pragmatic means of calibrating predictive models that requires a minimum amount of validation data and that performs well in practice. To evaluate the impact of miscalibration on utility and cost via clinical usefulness analyses. Materials and Methods Observational, retrospective cohort study with electronic health record data from 120,000 inpatient admissions at an urban, academic center in Manhattan. The primary outcome was thirty-day readmission for three causes: all-cause, congestive heart failure, and chronic coronary atherosclerotic disease. Predictive modeling was performed via L1-regularized logistic regression. Calibration methods were compared including Platt Scaling, Logistic Calibration, and Prevalence Adjustment. Performance of predictive modeling and calibration was assessed via discrimination (c-statistic), calibration (Spiegelhalter Z-statistic, Root Mean Square Error [RMSE] of binned predictions, Sanders and Murphy Resolutions of the Brier Score, Calibration Slope and Intercept), and clinical usefulness (utility terms represented as costs). The amount of validation data necessary to apply each calibration algorithm was also assessed. Results C-statistics by diagnosis ranged from 0.7 for all-cause readmission to 0.86 (0.78–0.93) for congestive heart failure. Logistic Calibration and Platt Scaling performed best and this difference required analyzing multiple metrics of calibration simultaneously, in particular Calibration Slopes and Intercepts. Clinical usefulness analyses provided optimal risk thresholds, which varied by reason for readmission, outcome prevalence, and calibration algorithm. Utility analyses also suggested maximum tolerable intervention costs, e.g. $1,720 for all-cause readmissions based on a published cost of readmission of $11,000. Conclusions Choice of calibration method depends on availability of validation data and on performance. Improperly calibrated models may contribute to higher costs of intervention as measured via clinical usefulness. Decision-makers must understand underlying utilities or costs inherent in the use-case at hand to assess usefulness and will obtain the optimal risk threshold to trigger intervention and intervention cost limits as a result.

show abstract

Calibration drift in regression and machine learning models for acute kidney injury

Cited by 203 publications

References 64 publications

Use of machine learning to analyse routinely collected intensive care unit data: a systematic review

Use of machine learning to analyse routinely collected intensive care unit data: a systematic review

A Path for Translation of Machine Learning Products into Healthcare Delivery

Beyond discrimination: A comparison of calibration methods and clinical usefulness of predictive models of readmission risk

Contact Info

Product

Resources

About