Multi-Modal Transformer for Accelerated MR Imaging

Feng, Chun-Mei; Yan, Yunlu; Chen, Geng; Xu, Yong; Shao, Ling; Fu, Huazhu

doi:10.48550/arxiv.2106.14248

Cited by 4 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Approaches in this category assume the availability of large MRI training datasets to train the ViT model. Feng et al [296] propose Transformerbased architecture, MTrans, for accelerated multi-modal MR imaging. The main component of MTrans is the cross attention module that extracts and fuses complementary features from the auxiliary modality to the target modality.…”

Section: Undersampled Mri Reconstructionmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, reconstruction, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

show abstract

Section: Undersampled Mri Reconstructionmentioning

confidence: 99%

Transformers in Medical Imaging: A Survey

Shamshad¹,

Khan²,

Zamir³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…That is because: (1) The similar patches widely exist in MR images, while most methods ignore this property and only adopt CNN to learn the local information, which leads to the failure in reconstructing the high-quality aliasing-free image. Recently, some methods [7,40] have adopted selfattention and transformer to learn global information, while the structure information is still missing in the recovered results. In addition, self-attention and transformer are timeconsuming and memory-consuming.…”

Section: Zp Unet Dudormentioning

confidence: 99%

“…However, these meth- ods separately processed the undersampled MRI in the Spatial and Frequency domains and cannot learn the local and global information, simultaneously. In contrast, to learn the global information, Wu et al [40] proposed a self-attention network for MR imaging and Feng et al [7] designed a transformer network for MR imaging. However, the selfattention [3,14,36] and transformer [21,33] have high computation complexity and occupy a huge number of GPU memory.…”

Section: Mri Reconstructionmentioning

confidence: 99%

Contrastive Learning for Local and Global Learning MRI Reconstruction

Yi¹,

Liu²,

Le³

et al. 2021

Preprint

View full text Add to dashboard Cite

Magnetic Resonance Imaging (MRI) is an important medical imaging modality, while it requires a long acquisition time. To reduce the acquisition time, various methods have been proposed. However, these methods failed to reconstruct images with a clear structure for two main reasons. Firstly, similar patches widely exist in MR images, while most previous deep learning-based methods ignore this property and only adopt CNN to learn local information. Secondly, the existing methods only use clear images to constrain the upper bound of the solution space, while the lower bound is not constrained, so that a better parameter of the network cannot be obtained. To address these problems, we propose a Contrastive Learning for Local and Global Learning MRI Reconstruction Network (CLGNet). Specifically, according to the Fourier theory, each value in the Fourier domain is calculated from all the values in Spatial domain. Therefore, we propose a Spatial and Fourier Layer (SFL) to simultaneously learn the local and global information in Spatial and Fourier domains. Moreover, compared with self-attention and transformer, the SFL has a stronger learning ability and can achieve better performance in less time. Based on the SFL, we design a Spatial and Fourier Residual block as the main component of our model. Meanwhile, to constrain the lower bound and upper bound of the solution space, we introduce contrastive learning, which can pull the result closer to the clear image and push the result further away from the undersampled image. Extensive experimental results on different datasets and acceleration rates demonstrate that the proposed CLGNet achieves new state-of-the-art results.

show abstract

“…To further improve the quality of imaging, Feng et al [37] enabled an end-to-end joint reconstruction and super-resolution. Feng et al [38] further advanced the model for these dual tasks by incorporating the model with task-specific novel cross-attention modules.…”

Section: Introductionmentioning

confidence: 99%

Swin Transformer for Fast MRI

Huang¹,

Fang²,

Wu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Magnetic resonance imaging (MRI) is an important non-invasive clinical tool that can produce high-resolution and reproducible images. However, a long scanning time is required for high-quality MR images, which leads to exhaustion and discomfort of patients, inducing more artefacts due to voluntary movements of the patients and involuntary physiological movements. To accelerate the scanning process, methods by k -space undersampling and deep learning based reconstruction have been popularised. This work introduced SwinMR, a novel Swin transformer based method for fast MRI reconstruction. The whole network consisted of an input module (IM), a feature extraction module (FEM) and an output module (OM). The IM and OM were 2D convolutional layers and the FEM was composed of a cascaded of residual Swin transformer blocks (RSTBs) and 2D convolutional layers. The RSTB

show abstract

Multi-Modal Transformer for Accelerated MR Imaging

Cited by 4 publications

References 39 publications

Transformers in Medical Imaging: A Survey

Transformers in Medical Imaging: A Survey

Contrastive Learning for Local and Global Learning MRI Reconstruction

Swin Transformer for Fast MRI

Contact Info

Product

Resources

About