Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital

Dziadzko, Mikhail; Novotny, Paul J.; Sloan, Jeff A.; Gajic, Ognjen; Herasevich, Vitaly; Mirhaji, Parsa; Wu, Yongwei; Gong, Michelle N.

doi:10.1186/s13054-018-2194-7

Cited by 44 publications

(49 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…19 The present study demonstrates that incorporation of intraoperative physiological time-series data improves predictive accuracy, discrimination, and precision, presumably by representing important intraoperative events and physiological changes that influence postoperative clinical trajectories and complications. Dziadzko et al 20 used a random forest model to predict mortality or the need for greater than 48 h of MV using EHR data from patients admitted to academic hospitals, achieving excellent discrimination (AUROC, 0.90), similar to MySurgeryRisk discrimination for MV for greater than 48 h (AUROC, 0.96) using both preoperative and intraoperative data. Therefore, the MySurgeryRisk PostOp extension takes another step toward clinical utility, maintaining autonomous function while improving accuracy, discrimination, and precision.…”

Section: Discussionmentioning

confidence: 99%

Added Value of Intraoperative Data for Predicting Postoperative Complications: The MySurgeryRisk PostOp Extension

Datta

Loftus

Ruppert

et al. 2020

Journal of Surgical Research

View full text Add to dashboard Cite

Background: Models that predict postoperative complications often ignore important intraoperative events and physiological changes. This study tested the hypothesis that accuracy, discrimination, and precision in predicting postoperative complications would improve when using both preoperative and intraoperative data input data compared with preoperative data alone. Methods: This retrospective cohort analysis included 43,943 adults undergoing 52,529 inpatient surgeries at a single institution during a 5-y period. Random forest machine learning models in the validated MySurgeryRisk platform made patient-level predictions for seven postoperative complications and mortality occurring during hospital admission using electronic health record data and patient neighborhood characteristics. For each outcome, one model trained with preoperative data alone; one model trained with both preoperative and intraoperative data. Models were compared by accuracy, discrimination (expressed as area under the receiver operating characteristic curve), precision (expressed as area under the precision-recall curve), and reclassification indices. Results: Machine learning models incorporating both preoperative and intraoperative data had greater accuracy, discrimination, and precision than models using preoperative data alone for predicting all seven postoperative complications (intensive care unit length of stay >48 h, mechanical ventilation >48 h, neurologic complications including delirium, cardiovascular complications, acute kidney injury, venous thromboembolism, and wound complications), and in-hospital mortality (accuracy: 88% versus 77%; area under the receiver operating characteristic curve: 0.93 versus 0.87; area under the precision-recall

show abstract

Section: Discussionmentioning

confidence: 99%

Added Value of Intraoperative Data for Predicting Postoperative Complications: The MySurgeryRisk PostOp Extension

Datta

Loftus

Ruppert

et al. 2020

Journal of Surgical Research

View full text Add to dashboard Cite

show abstract

“…16 of the studies performed an internal validation, where a proportion of the entire study dataset was used to develop the EWS (training set), with the remaining proportion was used for validation (validation set). 9,11,18,19,21,23,25,29,33,35,36,39,40,42,45,47,58 Varying proportion sizes were used for the validation set ranging from 25.0% to 100%. The other studies did an external validation with a study population different from that used to develop the EWS.…”

Section: Validation Datasetmentioning

confidence: 99%

“…The most commonly used were imputing a value from the last observation (6 studies), imputing a median value (3 studies), a combination of the last observation and median (5 studies) and imputing with a normal value (4 studies). There were also some sophisticated imputation methods, such as using random forest 18 and multiple imputation 24 .…”

Section: Handling Of Missing Valuesmentioning

confidence: 99%

Early warning score validation methodologies and performance metrics: A systematic review

Fang

Lim

Balakrishnan

2020

Preprint

View full text Add to dashboard Cite

Background Early warning scores (EWS) have been developed as clinical prognostication tools to identify acutely deteriorating patients. With recent advancements in machine learning, there has been a proliferation of studies that describe the development and validation of novel EWS. Systematic reviews of published studies which focus on evaluating performance of both well-established and novel EWS have shown conflicting conclusions. A possible reason for this is the lack of consistency in the validation methods used. In this review, we aim to examine the methodologies and performance metrics used in studies which describe EWS validation. Methods A systematic review of all eligible studies in the MEDLINE database from inception to 22-Feb-2019 was performed. Studies were eligible if they performed validation on at least one EWS and reported associations between EWS scores and mortality, intensive care unit (ICU) transfers, or cardiac arrest (CA) of adults within the inpatient setting. Two reviewers independently did a full-text review and performed data abstraction by using standardized data-worksheet based on the TRIPOD (Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) checklist. Meta-analysis was not performed due to heterogeneity. Results The key differences in validation methodologies identified were (1) validation population characteristics, (2) outcomes of interest, (3) case definition, intended time of use and aggregation methods, and (4) handling of missing values in the validation dataset. In terms of case definition, among the 34 eligible studies, 22 used the patient episode case definition while 10 used the observation set case definition, and 2 did the validation using both case definitions. Of those that used the patient episode case definition, 11 studies used a single point of time score to validate the EWS, most of which used the first recorded observation. There were also more than 10 different performance metrics reported among the studies. Conclusions Methodologies and performance metrics used in studies performing validation on EWS were not consistent hence making it difficult to interpret and compare EWS performance. Standardizing EWS validation methodology and reporting can potentially address this issue.

show abstract

“…[6][7][8] Several track and trigger systems (TTSs) using discrete numeric values such as vital signs and laboratory results are used in RRSs. [9,10] As conventional TTSs have limitations in detecting deterioration in patients, several researchers have adopted deep learning based algorithms to deal with these numeric values, which performed better than conventional tools. [11][12][13][14][15] However, the performances of these novel TTSs were also not satisfactory, and further improvement is needed to use the algorithms with electrical health records.…”

Section: Introductionmentioning

confidence: 99%

Artificial Intelligence Algorithm for Predicting Cardiac Arrest Using Electrocardiography

Kwon

Kim

Jeon

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: In-hospital cardiac arrest is a major burden in health care. Although several track-and-trigger systems are used to predict cardiac arrest, they often have unsatisfactory performances. We hypothesized that a deep-learning-based artificial intelligence algorithm (DLA) could effectively predict cardiac arrest using electrocardiography (ECG). We developed and validated a DLA for predicting cardiac arrest using ECG. Methods: We conducted a retrospective study that included 47,505 ECGs of 25,672 adult patients admitted to two hospitals, who underwent at least one ECG from October 2016 to September 2019. The endpoint was occurrence of cardiac arrest within 24 hours from ECG. Using subgroup analyses in patients who were initially classified as non-event, we confirmed the delayed occurrence of cardiac arrest and unexpected intensive care unit transfer over 14 days.Results: We used 32,294 ECGs of 10,461 patients and 4,483 ECGs of 4,483 patients from a hospital were used as development and internal validation data, respectively. Additionally, 10,728 ECGs of 10,728 patients from another hospital were used as external validation data, which confirmed the robustness of the developed DLA. During internal and external validation, the areas under the receiver operating characteristic curves of the DLA in predicting cardiac arrest within 24 hours were 0.913 and 0.948, respectively. The high risk group of the DLA showed a significantly higher hazard for delayed cardiac arrest (5.74% vs. 0.33%, P < 0.001) and unexpected intensive care unit transfer (4.23% vs. 0.82%, P < 0.001). A sensitivity map of the DLA displayed the ECG regions used to predict cardiac arrest, with the DLA focused most on the QRS complex. Conclusions: Our DLA successfully predicted cardiac arrest using diverse formats of ECG. The results indicate that cardiac arrest could be screened and predicted not only with a conventional 12-lead ECG, but also with a single-lead ECG using a wearable device that employs our DLA.

show abstract

Multicenter derivation and validation of an early warning score for acute respiratory failure or death in the hospital

Cited by 44 publications

References 34 publications

Added Value of Intraoperative Data for Predicting Postoperative Complications: The MySurgeryRisk PostOp Extension

Added Value of Intraoperative Data for Predicting Postoperative Complications: The MySurgeryRisk PostOp Extension

Early warning score validation methodologies and performance metrics: A systematic review

Artificial Intelligence Algorithm for Predicting Cardiac Arrest Using Electrocardiography

Contact Info

Product

Resources

About