Statistical Modeling and Aggregate‐Weighted Scoring Systems in Prediction of Mortality and ICU Transfer: A Systematic Review

Abstract

BACKGROUND:The clinical deterioration of patients in general hospital wards is an important safety issue. Aggregate-weighted early warning systems (EWSs) may not detect risk until patients present with acute decline. PURPOSE:We aimed to compare the prognostic test accuracy and clinical workloads generated by EWSs using statistical modeling (multivariable regression or machine learning) versus aggregate-weighted tools.DATA SOURCES: We searched PubMed and CINAHL using terms that described clinical deterioration and use of an advanced EWS. STUDY SELECTION:The outcome was clinical deterioration (intensive care unit transfer or death) of adult patients on general hospital wards. We included studies published from January 1, 2012 to September 15, 2018.DATA EXTRACTION: Following 2015 PRIMSA systematic review protocol guidelines; 2015 TRIPOD criteria for predictive model evaluation; and the Cochrane Collaboration guidelines, we reported model performance, adjusted positive predictive value (PPV), and conducted simulations of workup-to-detection ratios.DATA SYNTHESIS: Of 285 articles, six studies reported the model performance of advanced EWSs, and five were of high quality. All EWSs using statistical modeling identified at-risk patients with greater precision than aggregate-weighted EWSs (mean AUC 0.80 vs 0.73). EWSs using statistical modeling generated 4.9 alerts to find one true positive case versus 7.1 alerts in aggregateweighted EWSs; a nearly 50% relative workload increase for aggregate-weighted EWSs.CONCLUSIONS: Compared with aggregate-weighted tools, EWSs using statistical modeling consistently demonstrated superior prognostic performance and generated less workload to identify and treat one true positive case. A standardized approach to reporting EWS model performance is needed, including outcome definitions, pretest probability, observed and adjusted PPV, and workupto-detection ratio.

Improving early warning scores – more data, better validation, the same response

“…Without confirmation of acceptable discrimination and calibration, the predictive accuracy of a model cannot be stated firmly [8]. Early warning scores developed using statistical analysis of routinely collected data representative of clinical practice appear to provide both improved discrimination and calibration, with potential for better and more efficient care than is possible [9] using consensus-based scores, such as NEWS-2.…”

The optimal threshold for prompt clinical review: An external validation study of the national early warning score

“…Because the PPV is a function of the disease prevalence, the PPV of the second composite outcome (a + e + f) was even lower. (Linnen et al, 2019). Typically, studies reporting a comparison between an externally developed EWS and an in-house derived score report higher c-statistics in the latter.…”

“…The AUROCs calculated for the different outcomes in this study varied significantly (range 0.716–0.885). Studies comparing the performance of different EWSs yielded diverse results (Linnen et al., 2019). Typically, studies reporting a comparison between an externally developed EWS and an in‐house derived score report higher c‐statistics in the latter.…”

Machine Learning for Pulmonary and Critical Care Medicine: A Narrative Review

“…There were 135 papers duplicated between these two categories. Two systematic reviews were identified [18,19]. Given the narrative nature of this review, the final cohort of papers was hand-picked to provide the reader with the best general overview of the topic and was not meant to be comprehensive.…”