Generating Radiology Reports via Memory-driven Transformer

Chen, Zhihong; Yan, Shuicheng; Chang, Tsung-Hui; Wan, Xiang

doi:10.48550/arxiv.2010.16056

Cited by 21 publications

(46 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Examples include a memory-driven transformer to capture similar patterns in reports, uncertainty quantification for reliable report generation, a curriculum learning-based method, and an unsupervised approach to avoid paired training datasets. Chen et al [364] propose a memory-driven transformer to exploit similar patterns in the radiology image reports. Specifically, they add a module to each layer of transformerbased decoder by optimizing the original layer normalization with a novel memory-driven conditional layer normalization.…”

Section: Miscellaneousmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Miscellaneousmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Image-to-Text radiology report generation is a new type of image caption, which is an important application of natural language generation (NLG). It is to build assistive systems that take X-ray images of a patient and generate a textual report describing clinical observations in the images (Boag et al, 2020;Chen et al, 2020c;Jing et al, 2017;Li et al, 2018c;Liu et al, 2019a). This is a clinically important task, offering the potential to reduce radiologists' repetitive work and generally improve clinical communication (Kahn Jr et al, 2009).…”

Section: Image Captionmentioning

confidence: 99%

Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods

Li¹,

Wu²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Natural Language Generation (NLG) has made great progress in recent years due to the development of deep learning techniques such as pre-trained language models. This advancement has resulted in more fluent, coherent and even properties controllable (e.g. stylistic, sentiment, length etc.) generation, naturally leading to development in downstream tasks such as abstractive summarization, dialogue generation, machine translation, and data-to-text generation. However, the faithfulness problem that the generated text usually contains unfaithful or non-factual information has become the biggest challenge, which makes the performance of text generation unsatisfactory for practical applications in many real-world scenarios. Many studies on analysis, evaluation, and optimization methods for faithfulness problems have been proposed for various tasks, but have not been organized, compared and discussed in a combined manner. In this survey, we provide a systematic overview of the research progress on the faithfulness problem of NLG, including problem analysis, evaluation metrics and optimization methods. We organize the evaluation and optimization methods for different tasks into a unified taxonomy to facilitate comparison and learning across tasks. Several research trends are discussed further. NLG Fluency Grammatical Coherence InformativenessDiversity Specificity Redundnacy Controllability Stylistic Attributes Content Faithfulness Consistency Fidelity Factuality Figure 1: Four aspects of the NLG challenge. Faithfulness has become the biggest challenge in modern natural language generation.

show abstract

“…The Transformer was first introduced in [4,29], and developed in computer vision in 2021 [22]. Many works attempted to modify the architecture of ViT [30][31][32] for better performance or apply Transformer model into multidisciplinary researches [8,[33][34][35][36][37][38][39][40]. For example, a gated axial-attention model [36] was proposed to overcome the problem of lacking data samples in medical image segmentation.…”

Section: Related Workmentioning

confidence: 99%

POViT: Vision Transformer for Multi-objective Design and Characterization of Nanophotonic Devices

Chen¹,

Li²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

We solve a fundamental challenge in semiconductor IC design: the fast and accurate characterization of nanoscale photonic devices. Much like the fusion between AI and EDA, many efforts have been made to apply DNNs such as convolutional neural networks (CNN) to prototype and characterize next-gen optoelectronic devices commonly found in photonic integrated circuits (PICs). These prior works generally strive to predict the quality factor (Q) and modal volume (V) of for instance, photonic crystals, with ultra-high accuracy and speed. However, state-ofthe-art models are still far from being directly applicable in the real-world: e.g. the correlation coefficient of V (V coef f ) is only about 80%, which is much lower than what it takes to generate reliable and reproducible nanophotonic designs. Recently, attention-based transformer models have attracted extensive interests and been widely used in CV and NLP. In this work, we propose the first-ever Transformer model (POViT) to efficiently design and simulate semiconductor photonic devices with multiple objectives. Unlike the standard Vision Transformer (ViT), we supplied photonic crystals as data input and changed the activation layer from GELU [1] to an absolute-value function (ABS). Our experiments show that POViT exceeds results reported by previous models significantly. The correlation coefficient V coef f increases by over 12% (i.e., to 92.0%) and the prediction errors of Q is reduced by an order of magnitude, among several other key metric improvements. Our work has the potential to drive the expansion of EDA to fully automated photonic design. The complete dataset and code will be released to aid researchers endeavoring in the interdisciplinary field of physics and computer science 1 .Recently, Deep Learning (DL) have been widely utilized in multiple fields such as medical imaging [2,3], NLP [4,5], autonomous driving [6], face recognition [7] and object detection [8]. Neural

show abstract

Generating Radiology Reports via Memory-driven Transformer

Cited by 21 publications

References 25 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods

POViT: Vision Transformer for Multi-objective Design and Characterization of Nanophotonic Devices

Contact Info

Product

Resources

About