Derivation, Validation, Sustained Performance, and Clinical Impact of an Electronic Medical Record–Based Perioperative Delirium Risk Stratification Tool

Abstract

BACKGROUND: Postoperative delirium is an important problem for surgical inpatients and was the target of a multidisciplinary quality improvement project at our institution. We developed and tested a semiautomated delirium risk stratification instrument, Age, WORLD backwards, Orientation, iLlness severity, Surgery-specific risk (AWOL-S), in 3 independent cohorts from our tertiary care hospital and describe its performance characteristics and impact on clinical care. METHODS: The risk stratification instrument was derived with elective surgical patients who were admitted at least overnight and received at least 1 postoperative delirium screen (Nursing Delirium Screening Scale [NuDESC] or Confusion Assessment Method for the Intensive Care Unit [CAM-ICU]) and preoperative cognitive screening tests (orientation to place and ability to spell WORLD backward). Using data pragmatically collected between December 7, 2016, and June 15, 2017, we derived a logistic regression model predicting probability of delirium in the first 7 postoperative hospital days. A priori predictors included age, cognitive screening, illness severity or American Society of Anesthesiologists physical status, and surgical delirium risk. We applied model odds ratios to 2 subsequent cohorts (“validation” and “sustained performance”) and assessed performance using area under the receiver operator characteristic curves (AUC-ROC). A post hoc sensitivity analysis assessed performance in emergency and preadmitted patients. Finally, we retrospectively evaluated the use of benzodiazepines and anticholinergic medications in patients who screened at high risk for delirium. RESULTS: The logistic regression model used to derive odds ratios for the risk prediction tool included 2091 patients. Model AUC-ROC was 0.71 (0.67–0.75), compared with 0.65 (0.58–0.72) in the validation (n = 908) and 0.75 (0.71–0.78) in the sustained performance (n = 3168) cohorts. Sensitivity was approximately 75% in the derivation and sustained performance cohorts; specificity was approximately 59%. The AUC-ROC for emergency and preadmitted patients was 0.71 (0.67–0.75; n = 1301). After AWOL-S was implemented clinically, patients at high risk for delirium (n = 3630) had 21% (3%–36%) lower relative risk of receiving an anticholinergic medication perioperatively after controlling for secular trends. CONCLUSIONS: The AWOL-S delirium risk stratification tool has moderate accuracy for delirium prediction in a cohort of elective surgical patients, and performance is largely unchanged in emergent/preadmitted surgical patients. Using AWOL-S risk stratification as a part of a multidisciplinary delirium reduction intervention was associated with significantly lower rates of perioperative anticholinergic but not benzodiazepine, medications in those at high risk for delirium. AWOL-S offers a feasible starting point for electronic medical record–based postoperative delirium risk stratification and may serve as a useful paradigm for other institutions.

Machine Learning Algorithm Using Electronic Chart-Derived Data to Predict Delirium After Elderly Hip Fracture Surgeries: A Retrospective Case-Control Study

“…In this study, multivariate logistic regression yielded an AUC of 0.779 in postoperative delirium prediction, which was similar to previous studies (16). Existing predictive models such as the Vochteloo model, used a 9-item model to achieve a score adding formula, with a score of five as a cutoff point (17).…”

“…AWOL (age >79 years, failure to spell world backward, disorientation to place, and higher nurse-rated illness severity) model was another robust tool to predict delirium for newly admitted patients (18). Both models require extra questionnaires and are with moderate accuracy for delirium prediction, ranging from 65 to 75% (16).…”

Postoperative delirium prediction using machine learning models and preoperative electronic health record data

“…Many cases of POD can be prevented with multicomponent non-pharmacologic interventions [ 10 ], the Hospital Elder Life Program [ 11 ], or perioperative geriatric consultations [ 12 , 13 ]. Effective perioperative interventions combining delirium risk stratification with focused delirium prevention care practices have been described [ 14 ], though further improvement in the discrimination of the delirium risk stratification tool used in this intervention [ 15 ] could allow for better targeting of finite resources to the patients who need them most. Since preventative interventions require significant time and energy from busy clinicians [ 16 ], improving and automating risk stratification procedures is critically important.…”

Postoperative delirium: why, what, and how to confront it at your institution

“…Several such predictive tools have been published, such as the DEAR instrument, (22) the Delphi score (23), and the Age, World backward, Orientation, iLlness severity, and Surgery-specific risk (AWOL-S) score. (24) Communication of the patient's calculated POD risk to all stakeholders allows them to focus their attention and efforts. This communication can take many different forms, including, but not limited to, a flag in the electronic medical record (EMR), explicitly creating a new step in all provider handoffs, and providing written and video materials to families of high-risk patients.…”

“…The implementation phase included introduction of delirium risk stratification in the preoperative area, (24) modifications to the EMR to communicate high delirium risk to providers, focus on avoidance of Beers Criteria medications and best practice patient safety measures perioperatively, provider-to-provider communication, and non-pharmacologic delirium prevention measures in the recovery room. We particularly noted that the EMR is a powerful tool to help motivate behavior change.…”