Evaluating Outcome Prediction Models in Endovascular Stroke Treatment Using Baseline, Treatment, and Posttreatment Variables

Abstract

Background Statistical models to predict outcomes after endovascular therapy for acute ischemic stroke often incorporate baseline (pretreatment) variables only. We assessed the performance of stroke outcome prediction models for endovascular therapy in stroke in an iterative fashion using baseline, treatment‐related, and posttreatment variables. Methods Data from the ESCAPE‐NA1 (Safety and Efficacy of Nerinetide [NA‐1] in Subjects Undergoing Endovascular Thrombectomy for Stroke) trial were used to build 4 outcome prediction models using multivariable logistic regression: model 1 included baseline variables available before treatment decision making, model 2 included additional treatment‐related variables, model 3 additional posttreatment variables that become available early (within 24–48 hours), and model 4 later (beyond 48 hours) after endovascular therapy. The primary outcome was functional independence (90‐day Modified Rankin Scale score 0–2). Model performance was compared using the area under the receiver operating characteristic curve (AUC). Shapley values were used to determine marginal contributions of variables to outcome variance in the regression models. Results Among 1105 patients, functional independence was achieved by 666 (60.3%). When using baseline variables only (model 1), the AUC was 0.74 (95% CI, 0.71–0.77); this iteratively improved when treatment and posttreatment variables were added to the models (model 2: AUC, 0.77; 95% CI, 0.74–0.80; model 3: AUC, 0.80; 95% CI, 0.77–0.83; model 4: AUC, 0.82; 95% CI, 0.79–0.85). With baseline variables alone, 26% of patients who achieved functional independence were erroneously classified as not achieving functional independence. Even with the most comprehensive model, 19.8% of patients were misclassified as such. Patient age contributed most to outcome variance (Shapley value, 0.28), followed by severe adverse events including pneumonia (0.16) and intracranial hemorrhage at 24‐hours imaging (0.13). Conclusions A substantial contribution to outcomes after endovascular therapy comes from factors unrelated to currently collected baseline patient variables. One‐fifth of patients achieving functional independence were misclassified as not achieving independence, even with the most comprehensive model. Our findings suggest that the achievable accuracy of current outcome prediction models is limited, and caution should be used when applying them in clinical practice.

Multivariable Prediction Model for Futile Recanalization Therapies in Patients With Acute Ischemic Stroke

“…However, a recent study found that a large part of the variance in outcome after MT is explained by variables that are only known after the treatment decision has been made challenging the possibility to predict FRT in the emergency setting before knowing the outcome of the intervention -at least for MT patients. 48 Regarding the model performance, previous studies have shown that for most tabulated datasets, the differences in performance between different analytical machine learning methods and conventional logistic regression are minimal or non-existent. 21,29,37 Strenghts of this analysis include its large sample size with good quality data of predictors easily obtained in an emergency setting.…”

Association of Brain Atrophy With Functional Outcome and Recovery Trajectories After Thrombectomy

“…3 However, even when including the most comprehensive model (i.e., baseline + procedural + postprocedural variables), still one-fifth of patients were misclassified in post-EVT outcome. 4 Thus, exploring additional influencing factors, assessable at baseline, is crucial to refine prognostication and, at the same time, to inform clinical decision-making.…”