Comparison of Chest Radiograph Captions Based on Natural Language Processing vs Completed by Radiologists

Abstract

ImportanceArtificial intelligence (AI) can interpret abnormal signs in chest radiography (CXR) and generate captions, but a prospective study is needed to examine its practical value.ObjectiveTo prospectively compare natural language processing (NLP)-generated CXR captions and the diagnostic findings of radiologists.Design, Setting, and ParticipantsA multicenter diagnostic study was conducted. The training data set included CXR images and reports retrospectively collected from February 1, 2014, to February 28, 2018. The retrospective test data set included consecutive images and reports from April 1 to July 31, 2019. The prospective test data set included consecutive images and reports from May 1 to September 30, 2021.ExposuresA bidirectional encoder representation from a transformers model was used to extract language entities and relationships from unstructured CXR reports to establish 23 labels of abnormal signs to train convolutional neural networks. The participants in the prospective test group were randomly assigned to 1 of 3 different caption generation models: a normal template, NLP-generated captions, and rule-based captions based on convolutional neural networks. For each case, a resident drafted the report based on the randomly assigned captions and an experienced radiologist finalized the report blinded to the original captions. A total of 21 residents and 19 radiologists were involved.Main Outcomes and MeasuresTime to write reports based on different caption generation models.ResultsThe training data set consisted of 74 082 cases (39 254 [53.0%] women; mean [SD] age, 50.0 [17.1] years). In the retrospective (n = 8126; 4345 [53.5%] women; mean [SD] age, 47.9 [15.9] years) and prospective (n = 5091; 2416 [47.5%] women; mean [SD] age, 45.1 [15.6] years) test data sets, the mean (SD) area under the curve of abnormal signs was 0.87 (0.11) in the retrospective data set and 0.84 (0.09) in the prospective data set. The residents’ mean (SD) reporting time using the NLP-generated model was 283 (37) seconds—significantly shorter than the normal template (347 [58] seconds; P < .001) and the rule-based model (296 [46] seconds; P < .001). The NLP-generated captions showed the highest similarity to the final reports with a mean (SD) bilingual evaluation understudy score of 0.69 (0.24)—significantly higher than the normal template (0.37 [0.09]; P < .001) and the rule-based model (0.57 [0.19]; P < .001).Conclusions and RelevanceIn this diagnostic study of NLP-generated CXR captions, prior information provided by NLP was associated with greater efficiency in the reporting process, while maintaining good consistency with the findings of radiologists.

Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review

“…GenAI aids in diagnostic accuracy, although its focus on higher value creation in health care is limited. The articles in this review reported that they used deep learning (34/161, 21.1%) [ 49 , 59 , 60 , 62 , 63 , 65 , 68 , 71 , 79 , 89 , 100 , 107 , 108 , 111 , 115 , 123 , 125 , 130 - 145 ], machine learning (9/161, 5.6%) [ 53 , 55 , 83 , 91 , 110 , 146 - 149 ], and image analysis approaches of GenAI during the assistance process (13/161, 8.1%) [ 68 , 88 , 104 , 110 , 111 , 114 , 116 , 119 , 133 , 135 , 138 , 150 , 151 ]. Knowledge access using GenAI has the potential to enable more options and flexibility in serving patients.…”

“…Deep learning (34/161, 21.1%) [ 49 , 59 , 60 , 62 , 63 , 65 , 68 , 71 , 79 , 89 , 100 , 107 , 108 , 111 , 115 , 123 , 125 , 130 - 145 ]…”

Advances in research and application of artificial intelligence and radiomic predictive models based on intracranial aneurysm images

“…Additionally, an interactive radiology report diagnostic and evaluation system based on LLM can extend the detailed description of AI model detection results. For example, a study by Zhang et al ( 50 ) developed an AI captioning system based on the Transformer’s BERT model, which autonomously provides prior fields for lesion descriptions. This greatly enhances the model’s interpretability by providing textual descriptions based on lesion details and is expected to directly output diagnostic results.…”

Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

“…Artificial Intelligence (AI) is increasingly recognized as an important application in radiology [57,82,101,121]. In particular, the latest advancements in the creation and adaptation of multimodal foundation models (e.g., BioViL(-T) [8,17], ELIXR [137], MAIRA [58], Med-PaLM M [128]) invite high expectations of how the use of AI may transform clinical practice through efficiency and quality gains [121]; and improved overall patient care.…”

“…In this work, we focus particularly on the combination of large language models (LLMs) with vision capabilities -as so called vision-language models (VLMs). In the context of radiology imaging, this modality combination enables tasks such as: automatically generating a radiology report from a medical image (e.g., [57,58,148]); using text queries to answer questions about a radiology image (cf. [137]); or detecting errors in a radiology report text through its comparison with the image.…”