Automatic Determination of the Need for Intravenous Contrast in Musculoskeletal MRI Examinations Using IBM Watson’s Natural Language Processing Algorithm

Abstract

Magnetic resonance imaging (MRI) protocoling can be time- and resource-intensive, and protocols can often be suboptimal dependent upon the expertise or preferences of the protocoling radiologist. Providing a best-practice recommendation for an MRI protocol has the potential to improve efficiency and decrease the likelihood of a suboptimal or erroneous study. The goal of this study was to develop and validate a machine learning-based natural language classifier that can automatically assign the use of intravenous contrast for musculoskeletal MRI protocols based upon the free-text clinical indication of the study, thereby improving efficiency of the protocoling radiologist and potentially decreasing errors. We utilized a deep learning-based natural language classification system from IBM Watson, a question-answering supercomputer that gained fame after challenging the best human players on Jeopardy! in 2011. We compared this solution to a series of traditional machine learning-based natural language processing techniques that utilize a term-document frequency matrix. Each classifier was trained with 1240 MRI protocols plus their respective clinical indications and validated with a test set of 280. Ground truth of contrast assignment was obtained from the clinical record. For evaluation of inter-reader agreement, a blinded second reader radiologist analyzed all cases and determined contrast assignment based on only the free-text clinical indication. In the test set, Watson demonstrated overall accuracy of 83.2% when compared to the original protocol. This was similar to the overall accuracy of 80.2% achieved by an ensemble of eight traditional machine learning algorithms based on a term-document matrix. When compared to the second reader's contrast assignment, Watson achieved 88.6% agreement. When evaluating only the subset of cases where the original protocol and second reader were concordant (n = 251), agreement climbed further to 90.0%. The classifier was relatively robust to spelling and grammatical errors, which were frequent. Implementation of this automated MR contrast determination system as a clinical decision support tool may save considerable time and effort of the radiologist while potentially decreasing error rates, and require no change in order entry or workflow.

From Data to Value: How Artificial Intelligence Augments the Radiology Business to Create Value

“…In the test set, the IBM algorithm showed an overall accuracy of 83.2% when compared with the original protocol and 88.6% when compared with a blinded radiologist second reader. 26 A recent pilot study leveraged a computer-assisted detection (CAD) system and three-dimensional laboratory radiology technologists to assess CAD-processed T2-weighted fluid attenuated inversion recovery noncontrast images of the brain and spine to determine whether patients with multiple sclerosis should receive intravenous contrast. The study suggests a drastic reduction in contrast use, allowing for savings to the health care system from purchasing the contrast agent, time to administer contrast, and time for image generation and interpretation.…”

Artificial Intelligence in Musculoskeletal Imaging: Review of Current Literature, Challenges, and Trends

“…AI may support in ordering decisions by retrieving pertinent patient information including allergies to contrast media, MRI-sensitive devices, or implants that ideally are gathered automatically from an electronic radiologic request or the digital medical record, and by alerting the system to schedule the examination in the best conditions. [11][12][13][14] In musculoskeletal imaging, investigations for computed tomography (CT) or MRI in patients with metal hardware in the requested anatomical area should ideally be filtered by an electronic system, obtaining pertinent information from the request, from the latest radiograph, or from previous reports to schedule the patient automatically at dedicated scanners (e.g., dual-energy CT or 1.5-T MRI). Ideally, automated procedure selection algorithms are based on established guidelines such as American College of Radiology appropriateness criteria or European Society of Radiology iGuide, considering the efficiency, costs, and risks of various possible procedures.…”

“…For example, promising results were shown to determine automatically whether intravenous contrast was required in musculoskeletal MR investigations using IBM Watson natural language classifier on the free-text clinical indication of the study, reaching an accuracy of 83%. 13 This clinical decision support tool could help improve efficiency and decrease scheduling errors, among other advantages.…”

“…21 The analysis of clinical data could also be greatly improved by natural language-processing algorithms. 13 Finally, AI should help in the task of providing semiquantitative (i.e., the grading of certain pathologic findings such as lumbar canal stenosis, disk pathology, Kellgren-Lawrence grades on knee radiographs, or semiquantitative grading systems on knee MRIs, etc.) as well as quantitative analyses in reports.…”

“…Only very few reports have been published on how AI actually affects the workflow of a radiologist in daily practice. 13,14 Second, reproducing results of recently published studies is challenging, mainly because training data and the code to reproduce the investigation are rarely released. However, improvements have already been achieved in medical publications, and code release is now standard in engineering publications.…”