Automated detection of sudden unexpected death in epilepsy risk factors in electronic medical records using natural language processing

Abstract

Objective Sudden unexpected death in epilepsy (SUDEP) is an important cause of mortality in epilepsy. However, there is a gap in how often providers counsel patients about SUDEP. One potential solution is to electronically prompt clinicians to provide counseling via automated detection of risk factors in electronic medical records (EMRs). We evaluated (1) the feasibility and generalizability of using regular expressions to identify risk factors in EMRs and (2) barriers to generalizability. Methods Data included physician notes for 3000 patients from one medical center (home) and 1000 from five additional centers (away). Through chart review, we identified three SUDEP risk factors: (1) generalized tonic–clonic seizures, (2) refractory epilepsy, and (3) epilepsy surgery candidacy. Regular expressions of risk factors were manually created with home training data, and performance was evaluated with home test and away test data. Performance was evaluated by sensitivity, positive predictive value, and F‐measure. Generalizability was defined as an absolute decrease in performance by <0.10 for away versus home test data. To evaluate underlying barriers to generalizability, we identified causes of errors seen more often in away data than home data. To demonstrate how small revisions can improve generalizability, we removed three “boilerplate” standard text phrases from away notes and repeated performance. Results We observed high performance in home test data (F‐measure range = 0.86‐0.90), and low to high performance in away test data (F‐measure range = 0.53‐0.81). After removing three boilerplate phrases, away performance improved (F‐measure range = 0.79‐0.89) and generalizability was achieved for nearly all measures. The only significant barrier to generalizability was use of boilerplate phrases, causing 104 of 171 errors (61%) in away data. Significance Regular expressions are a feasible and probably a generalizable method to identify variables related to SUDEP risk. Our methods may be implemented to create large patient cohorts for research and to generate electronic prompts for SUDEP counseling.

Prospective validation of a machine learning model that uses provider notes to identify candidates for resective epilepsy surgery

“…Referral and evaluation of surgical candidacy include subjective criteria that are viewed differently between neurologists, such as the threshold for seizure frequency and severity needed to warrant a surgical recommendation . Our chart review was limited by this factor, as evidenced by our moderate interrater reliability, which was comparable to other epilepsy studies . However, our training set is less likely to be influenced by this bias.…”

“…These weights were chosen without incorporating domain knowledge, which allows for continued updating with little additional human effort. This is in contrast to other efforts that hand‐engineered a long list of regular expressions to construct epilepsy phenotypes, which would need to be updated over time …”

Investigation of bias in an epilepsy machine learning algorithm trained on physician notes

“…NLP can be used to evaluate epilepsy notes . We developed an NLP algorithm to assign surgical candidacy scores to patients based only on provider EHR notes .…”

“…NLP can be used to evaluate epilepsy notes. 12 We developed an NLP algorithm to assign surgical candidacy scores to patients based only on provider EHR notes. 13,14 This algorithm was incorporated into a pediatric hospital's EHR and is used to alert neurologists when they are scheduled to see a potential candidate for resective epilepsy surgery.…”

Can Big Data guide prognosis and clinical decisions in epilepsy?

“…ExECT was built on top of open source NLP software called GATE (General Architecture for Text Engineering), and achieved a precision of 91%. NLP tools have also been used for automatic detection of SUDEP risk factors from physician notes in EHRs, to generate electronic prompts for clinicians to counsel patients on SUDEP risk 37,62 . Three SUDEP risk factors were targeted: generalized tonic‐clonic seizures, refractory epilepsy, and epilepsy surgery candidacy.…”