Automated Outcome Classification of Emergency Department Computed Tomography Imaging Reports

Abstract

Background Reliably abstracting outcomes from free-text electronic medical records remains a challenge. While automated classification of free text has been a popular medical informatics topic, performance validation using real-world clinical data has been limited. The two main approaches are linguistic (natural language processing [NLP]) and statistical (machine learning). The authors have developed a hybrid system for abstracting computed tomography (CT) reports for specified outcomes. Objectives The objective was to measure performance of a hybrid NLP and machine learning system for automated outcome classification of emergency department (ED) CT imaging reports. The hypothesis was that such a system is comparable to medical personnel doing the data abstraction. Methods A secondary analysis was performed on a prior diagnostic imaging study on 3,710 blunt facial trauma victims. Staff radiologists dictated CT reports as free text, which were then deidentified. A trained data abstractor manually coded the reference standard outcome of acute orbital fracture, with a random subset double-coded for reliability. The data set was randomly split evenly into training and testing sets. Training patient reports were used as input to the Medical Language Extraction and Encoding (MedLEE) NLP tool to create structured output containing standardized medical terms and modifiers for certainty and temporal status. Findings were filtered for low certainty and past/future modifiers and then combined with the manual reference standard to generate decision tree classifiers using data mining tools Waikato Environment for Knowledge Analysis (WEKA) 3.7.5 and Salford Predictive Miner 6.6. Performance of decision tree classifiers was evaluated on the testing set with or without NLP processing. Results The performance of machine learning alone was comparable to prior NLP studies (sensitivity = 0.92, specificity = 0.93, precision = 0.95, recall = 0.93, f-score = 0.94), and the combined use of NLP and machine learning shows further improvement (sensitivity = 0.93, specificity = 0.97, precision = 0.97, recall = 0.96, f-score = 0.97). This performance is similar to, or better than, that of medical personnel in previous studies. Conclusions A hybrid NLP and machine learning automated classification system shows promise in coding free-text electronic clinical data.

Natural Language Processing Technologies in Radiology Research and Clinical Applications

“…Another powerful classification model that has become very popular in recent years is the support vector machine, which implicitly maps the features to a much higher dimensional space so as to derive many complex features automatically from the existing one, giving the model much better adaptivity. Both the maximum entropy and support vector machine models are often encountered in radiology NLP applications (22,24,(26)(27)(28)(29).…”

“…However, for certain tasks, the benefit may be less pronounced. For example, machine learning-based classification of acute orbital fractures in emergency department CT reports obtained in 3710 consecutive patients who presented with blunt orbital trauma was only slightly improved with use of linguistic NLP to extract features (sensitivity of 93.3% versus 92.5% and specificity of 96.9% versus 93.3%, respectively) (26).…”

“…Such use of NLP is of particular relevance to research because one could potentially develop a technique to preserve temporal information in the collective EMR data of each individual patient being de-identified. Other investigators have relied on manual or purpose-written software to achieve patient-linked de-identification (26).…”

Clinical Data Reuse or Secondary Use: Current Status and Potential Future Progress

“…Examples of the combined use of standard NLP and text-and data-mining are found in [139][140][141] where cTAKES is used with Boolean logic to perform phenotyping and to extract drug-side effects. MedLEE was applied for: 1) adverse drug reaction (ADR) signaling, where the association between a drug and an ADR was obtained by using disproportionality analysis [142,143] or Boolean logic [144], or by building and analyzing statistical distributions of concepts (i.e., diseases, symptoms, medications) extracted from the narrative text [145]; 2) EHR-data driven phenotyping using Boolean logic on MedLEE-extracted concepts [136,146]; 3) automated classification of outcomes from the analysis of emergency department computed tomography imaging reports using machine learning methods, such as decision trees [147]. MetaMap has been used with logistic regression in [148] to discover inappropriate use of emergency room based on information on drugs, psychological characteristics, diagnoses, and symptoms.…”

Automated information extraction from free-text EEG reports

“…“spikes” and “sharp waves”), generalized periodic discharges, lateralized periodic discharges, and rhythmic delta activity [1]. To derive useful knowledge from these free-text reports, researchers typically have to perform weeks to months of intensive manual work to review and categorize these reports [7] [8]. Automated algorithms offer the advantages of saving human labor, increased speed and the ability to scale the process to larger datasets [11].…”