Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Abstract

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.

Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation, and Diagnosis for COVID-19

“…2) The follow-up inside and outside of hospitals could be combined as a long period tracking for the COVID patients. 3) Multidisciplinary integration, i.e., medical imaging [85], natural language processing [86], and oncology and fusion [86], could benefit the overall follow-up procedure of measurement for COVID-19.…”

When Radiology Report Generation Meets Knowledge Graph

“…It has only one level of recurrent units in the decoder. We further extend the SAT model with additional sentence-level LSTM (SentSAT) (similar to the multi-level LSTM framework in (Yuan et al 2019) but without medical concept injection). The difference between SentSAT and our model is that the former uses attention over CNN features to obtain the context vector, while the latter first extracts chest abnormality graph features from the CNN features, propagates information on the graph, and then obtain the context vector using attention over graph node features.…”

“…Liu et al (Liu et al 2019) applied self-critical sequence training (Rennie et al 2017) based on reinforcement learning to optimize a clinically coherent reward, which focuses on the correct mention of disease keywords. Yuan et al (Yuan et al 2019) explored many ways of fusing frontal and lateral view features, and used attention over medical concepts which are extracted from Medical Text Indexer.…”

“…All rights reserved. report generation for chest x-ray images (Yuan et al 2019;Li et al 2019;Xue et al 2018;Jing et al 2017;Liu et al 2019). Most of the existing works based on the CNN-RNN Encoder-Decoder framework which has been widely applied in image captioning and visual question answering tasks.…”

Influence of AI on Detection of COVID-19

“…Apart from this, the need to integrate the data into machine learning models and accessibility of the data is also an uphill task. Under these circumstances, certain areas of worth mentioning for the datasets available are as listed below [14][15][16]:…”