Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays

Ni, Jianmo; Hsu, Chun‐Nan; Gentili, Amilcare; McAuley, Julian

doi:10.18653/v1/2020.findings-emnlp.176

“…Recent studies employed reinforcement learning to generate multiple summaries with varying lengths for a given text (Hyun et al. 2022) and to optimize factual consistency of generated summaries (Roit et al. 2023).…”

Section: Related Workmentioning

confidence: 99%

Abstractive summarization with deep reinforcement learning using semantic similarity rewards

Beken Fikri,

Oflazer,

Yanıkoğlu

2023

Nat. Lang. Eng.

1

0

View full text Add to dashboard Cite

Abstractive summarization is an approach to document summarization that is not limited to selecting sentences from the document but can generate new sentences as well. We address the two main challenges in abstractive summarization: how to evaluate the performance of a summarization model and what is a good training objective. We first introduce new evaluation measures based on the semantic similarity of the input and corresponding summary. The similarity scores are obtained by the fine-tuned BERTurk model using either the cross-encoder or a bi-encoder architecture. The fine-tuning is done on the Turkish Natural Language Inference and Semantic Textual Similarity benchmark datasets. We show that these measures have better correlations with human evaluations compared to Recall-Oriented Understudy for Gisting Evaluation (ROUGE) scores and BERTScore. We then introduce a deep reinforcement learning algorithm that uses the proposed semantic similarity measures as rewards, together with a mixed training objective, in order to generate more natural summaries in terms of human readability. We show that training with a mixed training objective function compared to only the maximum-likelihood objective improves similarity scores.

show abstract

“…Chest X-ray Report Generation Inspired by the success of deep learning models on image captioning, a lot of encoder-decoder based frameworks have been proposed (Jing et al, 2018(Jing et al, , 2019Liu et al, 2021bLiu et al, ,a, 2019cYuan et al, 2019;Xue et al, 2018;Li et al, 2018Zhang et al, 2020a;Kurisinkel et al, 2021;Ni et al, 2020;Nishino et al, 2020;Chen et al, 2020c;Wang et al, 2021;Boag et al, 2019;Syeda-Mahmood et al, 2020;Yang et al, 2020;Lovelace and Mortazavi, 2020;Zhang et al, 2020b;Miura et al, 2021). Specifically, Jing et al (2018) proposed a hierarchical LSTM with the attention mechanism (Bahdanau et al, 2015b;You et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Contrastive Attention for Automatic Chest X-ray Report Generation

Liu¹,

Yin²,

Wang³

et al. 2021

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021

View full text Add to dashboard Cite

Recently, chest X-ray report generation, which aims to automatically generate descriptions of given chest X-ray images, has received growing research interests. The key challenge of chest X-ray report generation is to accurately capture and describe the abnormal regions. In most cases, the normal regions dominate the entire chest X-ray image, and the corresponding descriptions of these normal regions dominate the final report. Due to such data bias, learning-based models may fail to attend to abnormal regions. In this work, to effectively capture and describe abnormal regions, we propose the Contrastive Attention (CA) model. Instead of solely focusing on the current input image, the CA model compares the current input image with normal images to distill the contrastive information. The acquired contrastive information can better represent the visual features of abnormal regions. According to the experiments on the public IU-X-ray and MIMIC-CXR datasets, incorporating our CA into several existing models can boost their performance across most metrics. In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis. Specifically, we achieve the state-ofthe-art results on the two public datasets.

show abstract

“…Chest X-ray Report Generation Inspired by the success of deep learning models on image captioning, a lot of encoder-decoder based frameworks have been proposed (Jing et al, 2018(Jing et al, , 2019Liu et al, 2021Liu et al, , 2019bYuan et al, 2019;Xue et al, 2018;Li et al, 2018Zhang et al, 2020a;Kurisinkel et al, 2021;Ni et al, 2020;Nishino et al, 2020;Chen et al, 2020c;Wang et al, 2021;Boag et al, 2019;Syeda-Mahmood et al, 2020;Yang et al, 2020;Lovelace and Mortazavi, 2020;Zhang et al, 2020b;Miura et al, 2021). Specifically, Jing et al (2018) proposed a hierarchical LSTM with the attention mechanism (Bahdanau et al, 2015b;You et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Contrastive Attention for Automatic Chest X-ray Report Generation

Liu

¹

,

Yin

²

,

Wang

³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, chest X-ray report generation, which aims to automatically generate descriptions of given chest X-ray images, has received growing research interests. The key challenge of chest X-ray report generation is to accurately capture and describe the abnormal regions. In most cases, the normal regions dominate the entire chest X-ray image, and the corresponding descriptions of these normal regions dominate the final report. Due to such data bias, learning-based models may fail to attend to abnormal regions. In this work, to effectively capture and describe abnormal regions, we propose the Contrastive Attention (CA) model. Instead of solely focusing on the current input image, the CA model compares the current input image with normal images to distill the contrastive information. The acquired contrastive information can better represent the visual features of abnormal regions. According to the experiments on the public IU-X-ray and MIMIC-CXR datasets, incorporating our CA into several existing models can boost their performance across most metrics. In addition, according to the analysis, the CA model can help existing models better attend to the abnormal regions and provide more accurate descriptions which are crucial for an interpretable diagnosis. Specifically, we achieve the state-ofthe-art results on the two public datasets.

show abstract

Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays

Cited by 12 publications

References 11 publications

Abstractive summarization with deep reinforcement learning using semantic similarity rewards

Abstractive summarization with deep reinforcement learning using semantic similarity rewards

Contrastive Attention for Automatic Chest X-ray Report Generation

Contrastive Attention for Automatic Chest X-ray Report Generation

Contact Info

Product

Resources

About