Automated Detection of Radiology Reports that Require Follow-up Imaging Using Natural Language Processing Feature Engineering and Machine Learning Classification

Abstract

While radiologists regularly issue follow-up recommendations, our preliminary research has shown that anywhere from 35 to 50% of patients who receive follow-up recommendations for findings of possible cancer on abdominopelvic imaging do not return for follow-up. As such, they remain at risk for adverse outcomes related to missed or delayed cancer diagnosis. In this study, we develop an algorithm to automatically detect free text radiology reports that have a follow-up recommendation using natural language processing (NLP) techniques and machine learning models. The data set used in this study consists of 6000 free text reports from the author's institution. NLP techniques are used to engineer 1500 features, which include the most informative unigrams, bigrams, and trigrams in the training corpus after performing tokenization and Porter stemming. On this data set, we train naive Bayes, decision tree, and maximum entropy models. The decision tree model, with an F1 score of 0.458 and accuracy of 0.862, outperforms both the naive Bayes (F1 score of 0.381) and maximum entropy (F1 score of 0.387) models. The models were analyzed to determine predictive features, with term frequency of n-grams such as "renal neoplasm" and "evalu with enhanc" being most predictive of a follow-up recommendation. Key to maximizing performance was feature engineering that extracts predictive information and appropriate selection of machine learning algorithms based on the feature set.

Deep Learning-Based Natural Language Processing in Radiology: The Impact of Report Complexity, Disease Prevalence, Dataset Size, and Algorithm Type on Model Performance

“…Our study and the referenced literature demonstrate the surprisingly high performance of deep learning NLP in radiology reporting. Information from both simple and more complex unstructured radiology reports can be extracted and used for downstream tasks such as epidemiological research, identification of incidental findings, assessment of diagnostic yield and imaging appropriateness, and labeling of images for training of computer vision algorithms [39][40][41][42].…”

Automatic detection of actionable radiology reports using bidirectional encoder representations from transformers

“…Some studies in Table 1 have the following features: (1) Multiple or all types of pathological entities are covered [ 7 – 15 ]. (2) The ground truth is based on clinical decisions, not just on the existence of specific expressions in radiology reports [ 16 – 18 ]. These two features can both lead to comprehensive detection of radiology reports with actionable findings.…”

“…Several research groups have investigated the automatic detection of actionable findings based on statistical machine learning [ 9 – 11 , 16 , 18 , 22 , 25 , 26 ]. However, these methods are mainly based on the frequency of words in each document, and other rich features such as word order and context are hardly taken into account.…”

Applications of Machine Learning Using Electronic Medical Records in Spine Surgery

“…For example, groups have used NLP across various medical disciplines to extract key clinical data from text documents such as patient records, 38 discharge summaries, [38][39][40] pathology reports, 40,41 and radiology reports. 40,42 These techniques can be adapted to assist spine surgeons via data extraction. For example, Tan et al 43 trained a model to identify information pertaining to low back pain (LBP) from lumbar spine imaging reports.…”