RadGraph: Extracting Clinical Entities and Relations from Radiology Reports

Abstract

Extracting structured clinical information from free-text radiology reports can enable the use of radiology report information for a variety of critical healthcare applications. In our work, we present RadGraph, a dataset of entities and relations in full-text chest X-ray radiology reports based on a novel information extraction schema we designed to structure radiology reports. We release a development dataset, which contains board-certified radiologist annotations for 500 radiology reports from the MIMIC-CXR dataset (14,579 entities and 10,889 relations), and a test dataset, which contains two independent sets of board-certified radiologist annotations for 100 radiology reports split equally across the MIMIC-CXR and CheXpert datasets. Using these datasets, we train and test a deep learning model, RadGraph Benchmark, that achieves a micro F1 of 0.82 and 0.73 on relation extraction on the MIMIC-CXR and CheXpert test sets respectively. Additionally, we release an inference dataset, which contains annotations automatically generated by RadGraph Benchmark across 220,763 MIMIC-CXR reports (around 6 million entities and 4 million relations) and 500 CheXpert reports (13,783 entities and 9,908 relations) with mappings to associated chest radiographs. Our freely available dataset can facilitate a wide range of research in medical natural language processing, as well as computer vision and multi-modal learning when linked to chest radiographs. * Equal Contribution † Equal Contribution Preprint. Under review.

Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation

“…We constructed metric-oracle reports for four metrics. These include BLEU 27 , BERTScore 28 , CheXbert vector similarity (s_emb) 13 and a novel metric RadGraph 23 F1. BLUE and BERTScore are general natural language metrics for measuring the similarity between machine-generated and human-generated texts.…”

“…The design of automated evaluation metrics that are aligned with manual expert evaluation has been a challenge for research in radiology report generation as well as medical report generation as a whole. Prior works have used metrics designed to improve upon n-gram matching [27][28][29][30][31] or include clinical awareness 12,13,15,23,17 , such as with BLEU 27 and CheXpert labels 12 . However, these evaluations nevertheless poorly approximate radiologists' evaluation of reports.…”

“…Since radiology reports are a special form of structured text that communicates diagnostics information, their quality depends highly on the correctness of clinical objects and descriptions, which is not a focus of traditional natural language metrics. To address this gap, the CheXbert labeler (which is improved from the CheXpert labeler) 12,13 and RadGraph 23 , were developed to parse radiology reports. We investigated whether they could be used as clinically aware metrics.…”

“…In this study, we quantitatively examine the correlation between automated metrics and the scoring of reports by radiologists. We propose a new automatic metric which computes the overlap in clinical entities and relations between a machine-generated report and a radiologist-generated report, called RadGraph 23 F1. We develop a methodology to predict a radiologist-determined error score from a combination of automated metrics, called RadCliQ.…”

A Medical Semantic-Assisted Transformer for Radiographic Report Generation

“…In order to train the Medical Concepts Generation Network (MCGN), we utilize RadGraph [9] to extract the pseudo-medical concepts as the ground-truth for multi-label classification (MLC). Specifically, the RadGraph is a knowledge graph of clinic radiology entitles and relations based on full-text chest x-ray radiology reports.…”