Longitudinal validation of an electronic health record delirium prediction model applied at admission in COVID-19 patients

Castro, Víctor M.; Hart, Kamber L.; Sacks, Chana A.; Murphy, Shawn N.; Perlis, Roy H.; McCoy, Thomas H.

doi:10.1016/j.genhosppsych.2021.10.005

Cited by 9 publications

(7 citation statements)

References 131 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The search strategy identified a total of 921 records; after duplicate removal and title and abstract screening, 114 full-text studies were retrieved, of which 3916–54 met the selection criteria for inclusion in the final analysis (figure 2).…”

Section: Resultsmentioning

confidence: 99%

“…Study characteristics are summarised in online supplemental table 1. Studies originated from the USA (n=12),17 19–23 25 41 43 50 51 54 Austria (n=9),24 28–31 33 39 47 48 China (n=6),26 32 35 49 52 53 Germany (n=3),37 45 46 South Korea (n=3),27 40 44 Canada (n=3),30 36 38 Brazil (n=1),16 Japan (n=1),34 Spain (n=1)18 and one study was labelled as international 42. Over the 6-year distribution of publications to June 2022, most studies were published in 2021 (n=10) and the first half of 2022 (n=12), indicating considerable growth in research in this area since the publication of previous reviews of studies published up to 2019 11 12.…”

Section: Resultsmentioning

confidence: 99%

“…Thirty of thirty-nine publications described the training and internal validation of a delirium model,17 18 21–26 30 32–41 43 44 46–54 with investigators of 6 of these studies (20%) externally validating their model in a subsequent paper 16 19 20 27 29 42. Investigators of three studies (10%) implemented and evaluated their model in real-time clinical workflows,2 8 31 45 but no publications described monitoring or optimising a deployed model.…”

Section: Resultsmentioning

confidence: 99%

“…Second, similar to the findings of previous reviews,11 12 most models described in our scoping review did not mature past the stage of internal validation. Only six studies validated their model on an external dataset16 19 20 27 29 42 despite evidence that models that perform well on ‘hold out’ training data usually have lower performance when applied to more noisy datasets from different institutions due to model overfitting 5…”

Section: Discussionmentioning

confidence: 99%

See 3 more Smart Citations

Navigating the machine learning pipeline: a scoping review of inpatient delirium prediction models

Strating

Hanjani

Tornvall

et al. 2023

BMJ Health Care Inform

View full text Add to dashboard Cite

ObjectivesEarly identification of inpatients at risk of developing delirium and implementing preventive measures could avoid up to 40% of delirium cases. Machine learning (ML)-based prediction models may enable risk stratification and targeted intervention, but establishing their current evolutionary status requires a scoping review of recent literature.MethodsWe searched ten databases up to June 2022 for studies of ML-based delirium prediction models. Eligible criteria comprised: use of at least one ML prediction method in an adult hospital inpatient population; published in English; reporting at least one performance measure (area under receiver-operator curve (AUROC), sensitivity, specificity, positive or negative predictive value). Included models were categorised by their stage of maturation and assessed for performance, utility and user acceptance in clinical practice.ResultsAmong 921 screened studies, 39 met eligibility criteria. In-silico performance was consistently high (median AUROC: 0.85); however, only six articles (15.4%) reported external validation, revealing degraded performance (median AUROC: 0.75). Three studies (7.7%) of models deployed within clinical workflows reported high accuracy (median AUROC: 0.92) and high user acceptance.DiscussionML models have potential to identify inpatients at risk of developing delirium before symptom onset. However, few models were externally validated and even fewer underwent prospective evaluation in clinical settings.ConclusionThis review confirms a rapidly growing body of research into using ML for predicting delirium risk in hospital settings. Our findings offer insights for both developers and clinicians into strengths and limitations of current ML delirium prediction applications aiming to support but not usurp clinician decision-making.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Navigating the machine learning pipeline: a scoping review of inpatient delirium prediction models

Strating

Hanjani

Tornvall

et al. 2023

BMJ Health Care Inform

View full text Add to dashboard Cite

show abstract

“…Delirium discharges were identified based on the Procedure Coding System from the International Statistical Classification of Diseases, Tenth Revision, Clinical Modification (ICD‐10‐CM/PCS). In order to make this pediatric analysis as comparable as possible to prior studies utilizing billing records to assess the incidence of delirium in adults, 5,33–35 delirium was defined based off the ICD‐10‐CM codes F01.51 (vascular dementia with behavioral disturbance), F03.90 (unspecified dementia without behavioral disturbance), F05 (delirium due to known physiological condition), F06.0 (psychotic disorder with hallucinations due to known physiological condition), F06.1 (catatonic disorder due to known physiological condition), F06.2 (psychotic disorder with delusions due to known physiological condition), F06.30 (mood disorder due to known physiological condition, unspecified), F06.4 (anxiety disorder due to known physiological condition), F06.8 (other specified mental disorders due to known physiological condition), F10.231 (alcohol dependence with withdrawal delirium), F15.920 (other stimulant use, unspecified with intoxication, uncomplicated), F19.921 (other psychoactive substance use, unspecified with intoxication with delirium), F19.939 (other psychoactive substance use, unspecified with withdrawal, unspecified), F19.950 (other psychoactive substance use, unspecified with psychoactive substance‐induced psychotic disorder with delusions), F19.951 (other psychoactive substance use, unspecified with psychoactive substance‐induced psychotic disorder with hallucinations), F19.97 (other psychoactive substance use, unspecified with psychoactive substance‐induced persisting dementia), G92 (toxic encephalopathy), G93.40 (encephalopathy, unspecified), G93.41 (metabolic encephalopathy), G93.49 (other encephalopathy), R40.0 (somnolence), R40.4 (transient alteration of awareness), and R41.82 (altered mental status, unspecified). Hospitalizations were included in the sample if any of these codes was among the discharge diagnosis list.…”

Section: Methodsmentioning

confidence: 99%

The occurrence of delirium diagnosis among youth hospitalizations in the United States: A Kids' Inpatient Database analysis

Luccarelli

Kalluri

Thom

et al. 2022

Acta Psychiatr Scand

Self Cite

View full text Add to dashboard Cite

Objectives: Delirium is an acute neuropsychiatric condition associated with increased morbidity and mortality. There is increasing recognition of delirium as a substantial health burden in younger patients, although few studies have characterized its occurrence. This study analyzes the occurrence of delirium diagnosis, its comorbidities, and cost among youth hospitalized in the United States.Methods: The Kids' Inpatient Database, a national all-payers sample of pediatric hospitalizations in general hospitals, was examined for the year 2019.Hospitalizations with a discharge diagnosis of delirium among patients aged 1-20 years were included in the analysis. Results: Delirium was diagnosed in 43,138 hospitalizations (95% CI: 41,170-45,106), or 2.3% of studied hospitalizations. Delirium was diagnosed in a broad range of illnesses, with suicide and self-inflicted injury as the most common primary discharge diagnosis among patients with delirium. In-hospital mortality was seven times greater in hospitalizations caring a delirium diagnosis. The diagnosis of delirium was associated with an adjusted increased hospital cost of $8648 per hospitalization, or $373 million in aggregate cost. Conclusions: Based on a large national claims database, delirium was diagnosed in youth at a lower rate than expected based on prospective studies. The relative absence of delirium diagnosis in claims data may reflect underdiagnosis, a failure to code, and/or a lower rate of delirium in general hospitals compared with other settings. Further research is needed to better characterize the incidence and prevalence of delirium in young people in the hospital setting.

show abstract

The Multidimensional Prognostic Index predicts incident delirium among hospitalized older patients with COVID-19: a multicenter prospective European study

Morganti,

Custodero,

Veronese

et al. 2024

Eur Geriatr Med

View full text Add to dashboard Cite

Purpose Incident delirium is a frequent complication among hospitalized older people with COVID-19, associated with increased length of hospital stay, higher morbidity and mortality rates. Although delirium is preventable with early detection, systematic assessment methods and predictive models are not universally defined, thus delirium is often underrated. In this study, we tested the role of the Multidimensional Prognostic Index (MPI), a prognostic tool based on Comprehensive Geriatric Assessment, to predict the risk of incident delirium. Methods Hospitalized older patients (≥ 65 years) with COVID-19 infection were enrolled (n = 502) from ten centers across Europe. At hospital admission, the MPI was administered to all the patients and two already validated delirium prediction models were computed (AWOL delirium risk-stratification score and Martinez model). Delirium occurrence during hospitalization was ascertained using the 4A’s Test (4AT). Accuracy of the MPI and the other delirium predictive models was assessed through logistic regression models and the area under the curve (AUC). Results We analyzed 293 patients without delirium at hospital admission. Of them 33 (11.3%) developed delirium during hospitalization. Higher MPI score at admission (higher multidimensional frailty) was associated with higher risk of incident delirium also adjusting for the other delirium predictive models and COVID-19 severity (OR = 12.72, 95% CI = 2.11–76.86 for MPI-2 vs MPI-1, and OR = 33.44, 95% CI = 4.55–146.61 for MPI-3 vs MPI-1). The MPI showed good accuracy in predicting incident delirium (AUC = 0.71) also superior to AWOL tool, (AUC = 0.63) and Martinez model (AUC = 0.61) (p < 0.0001 for both comparisons). Conclusions The MPI is a sensitive tool for early identification of older patients with incident delirium.

show abstract

Longitudinal validation of an electronic health record delirium prediction model applied at admission in COVID-19 patients

Cited by 9 publications

References 131 publications

Navigating the machine learning pipeline: a scoping review of inpatient delirium prediction models

Navigating the machine learning pipeline: a scoping review of inpatient delirium prediction models

The occurrence of delirium diagnosis among youth hospitalizations in the United States: A Kids' Inpatient Database analysis

The Multidimensional Prognostic Index predicts incident delirium among hospitalized older patients with COVID-19: a multicenter prospective European study

Contact Info

Product

Resources

About