A Medical Semantic-Assisted Transformer for Radiographic Report Generation

Abstract

Automated radiographic report generation is a challenging cross-domain task that aims to automatically generate accurate and semantic-coherence reports to describe medical images. Despite the recent progress in this field, there are still many challenges at least in the following aspects. First, radiographic images are very similar to each other, and thus it is difficult to capture the fine-grained visual differences using CNN as the visual feature extractor like many existing methods. Further, semantic information has been widely applied to boost the performance of generation tasks (e.g. image captioning), but existing methods often fail to provide effective medical semantic features. Toward solving those problems, in this paper, we propose a memoryaugmented sparse attention block utilizing bilinear pooling to capture the higher-order interactions between the input fine-grained image features while producing sparse attention. Moreover, we introduce a novel Medical Concepts Generation Network (MCGN) to predict fine-grained semantic concepts and incorporate them into the report generation process as guidance. Our proposed method shows promising performance on the recently released largest benchmark MIMIC-CXR. It outperforms multiple state-of-the-art methods in image captioning and medical report generation.

Style-Aware Radiology Report Generation with RadGraph and Few-Shot Prompting

“…(Yang et al, 2022) adopts the knowledge graph provided by RadGraph as general knowledge embedding. (Wang et al, 2022b) employed a classification loss for medical concepts provided by RadGraph. (Delbrouck et al, 2022) improves the factual completeness and correctness of generated radiology reports with a well-designed RadGraph reward.…”

Normal-Abnormal Decoupling Memory for Medical Report Generation

“…Additionally, presented a framework that leverages both a priori and a posteriori data to enhance generative reporting. (Wang et al, 2022c) utilized semantic extraction to improve generation, they overlooked the problems of noise and data bias prevalent in the reports. In (Jing et al, 2019), the data bias problem is considered, where the noise of normal statements still affects the optimization of the model despite the separate generation of normal/abnormal statements.…”

“…S&T (Vinyals et al, 2015), AdaAtt (Lu et al, 2017), Top-Down (Anderson et al, 2018), and the ones proposed for the medical domain, e.g. R2Gen (Chen et al, 2020), PPKED , M2TR (Nooralahzadeh et al, 2021), R2GenCMN (Chen et al, 2021b), XProNet (Wang et al, 2022a), GS-KET , R2GenRL (Qin and Song, 2022) and MSAT (Wang et al, 2022c).The results on S&T (Vinyals et al, 2015), AdaAtt (Lu et al, 2017), TopDown (Anderson et al, 2018) from (Chen et al, 2020), and the rest of the results were cited from the original paper. Table 1 shows the NLG metrics and Table 2 shows the CE metrics.…”

“…S&T (Vinyals et al, 2015), AdaAtt (Lu et al, 2017), and the ones proposed for the medical domain, e.g. R2Gen (Chen et al, 2020), PPKED , M2TR (Nooralahzadeh et al, 2021), R2GenCMN (Chen et al, 2021b) GSKET , R2GenRL (Qin and Song, 2022) and MSAT (Wang et al, 2022c). The results on S&T (Vinyals et al, 2015), AdaAtt (Lu et al, 2017) from (Chen et al, 2020), and the rest of the results were cited from the original paper.As we can see from the results in Table 7, our method can still achieve good results on the IU-Xray dataset, which indicates the generality of our method.…”

“…We argue that different words in the report possess varying levels of importance, with abnormal semantics being intuitively more significant than normal semantics. However, previous approaches (Jing et al, 2017;Yuan et al, 2019;Yin et al, 2019;Najdenkoska et al, 2021;Chen et al, 2021b;Wang et al, 2022c) tend to treat all report words equally. Therefore, we divide the reported words into four categories: normal/abnormal sentences and normal/abnormal semantics.…”