Prognostic Models for Traumatic Brain Injury Have Good Discrimination but Poor Overall Model Performance for Predicting Mortality and Unfavorable Outcomes

Abstract

BACKGROUND:The most extensively validated prognostic models for traumatic brain injury (TBI) are the Corticoid Randomization after Significant Head Injury (CRASH) and International Mission on Prognosis and Analysis of Clinical Trials (IMPACT). Model characteristics outside of area under the curve (AUC) are rarely reported. OBJECTIVE: To report the discriminative validity and overall model performance of the CRASH and IMPACT models for prognosticating death at 14 days (CRASH) and 6 months (IMPACT) and unfavorable outcomes at 6 months after TBI. METHODS: This retrospective cohort study included prospectively collected patients with severe TBI treated at a single level I trauma center (n = 467). CRASH and IMPACT percent risk values for the given outcome were computed. Unfavorable outcome was defined as a Glasgow Outcome Scale-Extended score of 1 to 4 at 6 months. Binary logistic regressions and receiver operating characteristic analyses were used to differentiate patients from the CRASH and IMPACT prognostic models. RESULTS: All models had low R 2 values (0.17-0.23) with AUC values from 0.77 to 0.81 and overall accuracies ranging from 72.4% to 78.3%. Sensitivity (35.3-50.0) and positive predictive values (66.7-69.2) were poor in the CRASH models, while specificity (52.3-53.1) and negative predictive values (58.1-63.6) were poor in IMPACT models. All models had unacceptable false positive rates (20.8%-33.3%). CONCLUSION: Our results were consistent with previous literature regarding discriminative validity (AUC = 0.77-0.81). However, accuracy and false positive rates of both the CRASH and IMPACT models were poor.

In Reply: Prognostic Models for Traumatic Brain Injury Have Good Discrimination But Poor Overall Model Performance for Predicting Mortality and Unfavorable Outcomes

“…We would like to thank Ms Ran and Dr Azad for their insightful comments. 1,2 Ms Ran and Dr Azad point out that the conclusions of our paper support those of the Commission on traumatic brain injury (TBI) report published in September 2022, namely that the Corticosteroid Randomization After Significant Head Injury (CRASH) and International Mission for Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) models are not suitable for clinical use and require reinvigoration with expanded clinical data points and modern analytics. As the authors point out, these prognostic models are up to 30 years old; current standards of TBI assessment, management, and rehabilitation have changed dramatically over that time period.…”

Letter: Prognostic Models for Traumatic Brain Injury Have Good Discrimination But Poor Overall Model Performance for Predicting Mortality and Unfavorable Outcomes

“…The authors leveraged a cohort of 467 patients with severe traumatic brain injury (TBI) at a level I trauma center to externally validate the Corticosteroid Randomization After Significant Head Injury (CRASH) and International Mission for Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) models for prediction of death and poor outcome after severe TBI at the individual patient level. 1 The authors found that both models have reasonable discrimination (ie, delineating patients with and without the outcome of interest) but poor accuracy and high false-positive rates. They urge caution when using these models to inform clinical decision-making.We are writing to discuss these results in the context of the recent update to the Commission on TBI published in Lancet Neurology titled "Traumatic brain injury: progress and challenges in prevention, clinical care, and research."…”

“…We read with great interest the publication titled “Prognostic Models for Traumatic Brain Injury Have Good Discrimination but Poor Overall Model Performance for Predicting Mortality and Unfavorable Outcomes” 1 and congratulate the authors on this important work. The authors leveraged a cohort of 467 patients with severe traumatic brain injury (TBI) at a level I trauma center to externally validate the Corticosteroid Randomization After Significant Head Injury (CRASH) and International Mission for Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) models for prediction of death and poor outcome after severe TBI at the individual patient level.…”

Performance of CRASH and IMPACT Prognostic Models for Traumatic Brain Injury at 12 and 24 Months Post-Injury

“…The discriminative validity for each model remained consistent over time and comparable to earlier recovery time points (AUC = 0.77–0.83). 16 Both models had poor fit for unfavorable outcomes ( R 2 = 0.15–0.23), explaining less than one quarter of the variation in outcomes for severe TBI patients. The CRASH model had significant values for the Hosmer-Lemeshow test at 12 and 24 months, indicating poor model calibration past 6 months post-TBI.…”