Multimodal Recurrent Model with Attention for Automated Radiology Report Generation

Xue, Yuan; Xu, Tao; Long, L. Rodney; Xue, Zhiyun; Antani, Sameer; Thoma, George R.; Huang, Xiaolei

doi:10.1007/978-3-030-00928-1_52

Cited by 128 publications

(99 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Chest Radiographic Observations: The task is formulated as a multi-label classification with 14 common radiographic observations following [5] including: enlarged cardiom, cardiomegaly, lung opacity, lung lesion, edema, consolidation, pneumonia, atelectasis, pneumothorax, pleural effusion, pleural other, fracture, support devices, and no finding. Compared with previous studies using pretrained encoders based on ImageNet [6,14], pretraining with images from the same domain yields better results. We add one full-connected layer as classifier and compute the binary cross entropy (BCE) loss.…”

Section: Image Encodermentioning

confidence: 64%

“…Radiology Report Generation: The evaluation metrics we use are BLEU [9], METEOR [2], and ROUGE [8] scores, all of which are widely used in image captioning and machine translation tasks. We compared the proposed model with several state-of-the-art baselines: (1) a visual attention based image captioning model (Vis-Att) [13]; (2) radiology report generation models, including a hierarchical decoder with co-attention (Co-Att) [6], multimodal generative model with visual attention (MM-Att) [14], and knowledge-drive retrieval based report generation (KERP) [7]; and (3) the proposed multi-view encoder with hierarchical decoder (MvH) model, the base model with visual attentions and early fusion (MvH+AttE), MvH with late fusion fashion (MvH+AttL), and the combination of late fusion with medical concepts (MvH+AttL+MC). MvH+AttL+MC* is an oracle run based on ground-truth medical concepts and considered as the upper bound of the improvement caused by applying medical concepts.…”

Section: Methodsmentioning

confidence: 99%

“…Additionally, we consider both frontal and lateral images of one patient as one input pair and enforce the prediction consistencies under different views by a mean square error (MSE) loss over the multi-view encoder outputs. The loss function is thus defined in Equation 1 where y i,j is the j-th ground truth entry (j ∈ [1,14])) of the i-th sample (i ∈ (1, N )), the frontal view and lateral view prediction are denoted asŷ i f and y i l . The encoder outputs global features after average-pooling, and local features v ∈ R k×dv from the last CNN block where k denotes the number of local regions and d v denotes the dimension.…”

Section: Image Encodermentioning

confidence: 99%

See 2 more Smart Citations

Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Yuan

Liao

Luo

et al. 2019

Lecture Notes in Computer Science

132

106

View full text Add to dashboard Cite

Generating radiology reports is time-consuming and requires extensive expertise in practice. Therefore, reliable automatic radiology report generation is highly desired to alleviate the workload. Although deep learning techniques have been successfully applied to image classification and image captioning tasks, radiology report generation remains challenging in regards to understanding and linking complicated medical visual contents with accurate natural language descriptions. In addition, the data scales of open-access datasets that contain paired medical images and reports remain very limited. To cope with these practical challenges, we propose a generative encoder-decoder model and focus on chest x-ray images and reports with the following improvements. First, we pretrain the encoder with a large number of chest x-ray images to accurately recognize 14 common radiographic observations, while taking advantage of the multi-view images by enforcing the cross-view consistency. Second, we synthesize multi-view visual features based on a sentence-level attention mechanism in a late fusion fashion. In addition, in order to enrich the decoder with descriptive semantics and enforce the correctness of the deterministic medical-related contents such as mentions of organs or diagnoses, we extract medical concepts based on the radiology reports in the training data and fine-tune the encoder to extract the most frequent medical concepts from the x-ray images. Such concepts are fused with each decoding step by a word-level attention model. The experimental results conducted on the Indiana University Chest X-Ray dataset demonstrate that the proposed model achieves the state-of-the-art performance compared with other baseline approaches.

show abstract

Section: Image Encodermentioning

confidence: 64%

Section: Methodsmentioning

confidence: 99%

Section: Image Encodermentioning

confidence: 99%

See 1 more Smart Citation

Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Yuan

Liao

Luo

et al. 2019

Lecture Notes in Computer Science

132

106

View full text Add to dashboard Cite

show abstract

“…Typically this involves conditioning a recurrent neural network (RNN) on image features encoded by a convolutional neural network (CNN). This method has shown great promise in non-specific image captioning tasks but has not generalized well to the complex domain of medical images [18,19].…”

Section: Introductionmentioning

confidence: 99%

“…For example, a pathologist must interpret a set of 8 different renal biopsy sections of the same patient to report the RDIF assay. Several methods have been proposed to enable captioning of multiple images by assuming images in the set exhibit temporal dependence [4] or contain multiple views/instances of the same object [18]. These assumptions are unsuitable for the multi-object temporally independent RDIF set.…”

Section: Introductionmentioning

confidence: 99%

CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

Maksoud

Wiliem

Zhao

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

This work tackles the problem of generating a medical report for multi-image panels. We apply our solution to the Renal Direct Immunofluorescence (RDIF) assay which requires a pathologist to generate a report based on observations across the eight different WSI in concert with existing clinical features. To this end, we propose a novel attention-based multi-modal generative recurrent neural network (RNN) architecture capable of dynamically sampling image data concurrently across the RDIF panel. The proposed methodology incorporates text from the clinical notes of the requesting physician to regulate the output of the network to align with the overall clinical context. In addition, we found the importance of regularizing the attention weights for word generation processes. This is because the system can ignore the attention mechanism by assigning equal weights for all members. Thus, we propose two regularizations which force the system to utilize the attention mechanism. Experiments on our novel collection of RDIF WSIs provided by a large clinical laboratory demonstrate that our framework offers significant improvements over existing methods.

show abstract

Comparison of Chest Radiograph Captions Based on Natural Language Processing vs Completed by Radiologists

et al. 2023

View full text Add to dashboard Cite

ImportanceArtificial intelligence (AI) can interpret abnormal signs in chest radiography (CXR) and generate captions, but a prospective study is needed to examine its practical value.ObjectiveTo prospectively compare natural language processing (NLP)-generated CXR captions and the diagnostic findings of radiologists.Design, Setting, and ParticipantsA multicenter diagnostic study was conducted. The training data set included CXR images and reports retrospectively collected from February 1, 2014, to February 28, 2018. The retrospective test data set included consecutive images and reports from April 1 to July 31, 2019. The prospective test data set included consecutive images and reports from May 1 to September 30, 2021.ExposuresA bidirectional encoder representation from a transformers model was used to extract language entities and relationships from unstructured CXR reports to establish 23 labels of abnormal signs to train convolutional neural networks. The participants in the prospective test group were randomly assigned to 1 of 3 different caption generation models: a normal template, NLP-generated captions, and rule-based captions based on convolutional neural networks. For each case, a resident drafted the report based on the randomly assigned captions and an experienced radiologist finalized the report blinded to the original captions. A total of 21 residents and 19 radiologists were involved.Main Outcomes and MeasuresTime to write reports based on different caption generation models.ResultsThe training data set consisted of 74 082 cases (39 254 [53.0%] women; mean [SD] age, 50.0 [17.1] years). In the retrospective (n = 8126; 4345 [53.5%] women; mean [SD] age, 47.9 [15.9] years) and prospective (n = 5091; 2416 [47.5%] women; mean [SD] age, 45.1 [15.6] years) test data sets, the mean (SD) area under the curve of abnormal signs was 0.87 (0.11) in the retrospective data set and 0.84 (0.09) in the prospective data set. The residents’ mean (SD) reporting time using the NLP-generated model was 283 (37) seconds—significantly shorter than the normal template (347 [58] seconds; P &lt; .001) and the rule-based model (296 [46] seconds; P &lt; .001). The NLP-generated captions showed the highest similarity to the final reports with a mean (SD) bilingual evaluation understudy score of 0.69 (0.24)—significantly higher than the normal template (0.37 [0.09]; P &lt; .001) and the rule-based model (0.57 [0.19]; P &lt; .001).Conclusions and RelevanceIn this diagnostic study of NLP-generated CXR captions, prior information provided by NLP was associated with greater efficiency in the reporting process, while maintaining good consistency with the findings of radiologists.

show abstract

Multimodal Recurrent Model with Attention for Automated Radiology Report Generation

Cited by 128 publications

References 16 publications

Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

Automatic Radiology Report Generation Based on Multi-view Image Fusion and Medical Concept Enrichment

CORAL8: Concurrent Object Regression for Area Localization in Medical Image Panels

Comparison of Chest Radiograph Captions Based on Natural Language Processing vs Completed by Radiologists

Contact Info

Product

Resources

About