A unified hybrid transformer for joint MRI sequences super-resolution and missing data imputation

Abstract

Objective: High-resolution (HR) multi-modal magnetic resonance imaging (MRI) is crucial in clinical practice for accurate diagnosis and treatment. However, challenges such as budget constraints, potential contrast agent deposition, and image corruption often limit the acquisition of multiple sequences from a single patient. Therefore, the development of novel methods to reconstruct under-sampled images and synthesize missing sequences is crucial for clinical and research applications. Approach: In this paper, we propose a unified hybrid framework called SIFormer, which utilizes any available low-resolution (LR) MRI contrast configurations to complete super-resolution (SR) of poor-quality MR images and impute missing sequences simultaneously in one forward process. SIFormer consists of a hybrid generator and a convolution-based discriminator. The generator incorporates two key blocks. First, the dual branch attention (DBA) block combines the long-range dependency building capability of the transformer with the high-frequency local information capture capability of the convolutional neural network (CNN) in a channel-wise split manner. Second, we introduce a learnable gating adaptation multi-layer perception (GA MLP) in the feed-forward block to optimize information transmission efficiently. Main Results: Comparative evaluations against six state-of-the-art methods demonstrate that SIFormer achieves enhanced quantitative performance and produces more visually pleasing results for image SR and synthesis tasks across multiple datasets. Significance: Extensive experiments conducted on multi-center multi-contrast MRI datasets, including both healthy individuals and brain tumor patients, highlight the potential of our proposed method to serve as a valuable supplement to MRI sequence acquisition in clinical and research settings.

Generative adversarial network-based synthesis of contrast-enhanced MR images from precontrast images for predicting histological characteristics in breast cancer

“…The synthesis of diagnostic images from noncontrast sequences was performed on brain MR images (Wang et al , 2023. A related work performed a preliminary study by simulating postcontrast images from precontrast images (Chung et al 2023).…”

Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry

“…Li et al [91] McMRSR 2D fastMRI [92] MRI MRI superresolution RMSE; SSIM; PSNR Multi-scale contextual matching Wang et al [93] SIFormer 2D IXI; BraTS [94] MRI MRI superresolution DICE; SSIM; PSNR Super-resolution with sequential MRI Forigua et al [95] SuperFormer 3D HCP [89] MRI MRI superresolution NRMSE; SSIM; PSNR 3D ViT Wu et al [96] KTMR 2D HCP [89] MRI Image denoising SSIM; PSNR Reconstruction in the context of k-space consistency Huang et al [97] ST-GAN 2D HCP [89] MRI Image denoising SSIM; PSNR; FID GAN based Swin Transformer Wang et al [98] TED-Net 2D NIH-AAPM [99] CT Image denoising MSE; SSIM; PSNR LDCT denoising Luthra et al [100] Eformer 2D NIH-AAPM [99] CT Image denoising RMSE; SSIM; PSNR LDCT denoising with residual learning Zhang et al [101] TransCT 2D NIH-AAPM [99] CT Image denoising MSE; SSIM; PSNR LDCT denoising with high & low frequency decomposition Guo et al [102] ReconFormer 2D fastMRI [92] MRI Fast MR imaging MSE; SSIM; PSNR Multi-scale learning & lightweight Zhou et al [103] DSFormer 2D IXI MRI Fast MR imaging MSE; SSIM; PSNR Self-supervised for multi-contrast MRI reconstruction Lyu et al [104] DuDoCAF 2D FastMRI [92] MRI Fast MR imaging MSE; SSIM; PSNR Recurrent Transformer for fast multicontrast MR imaging Korkmaz et al [105] SLATER 3D IXI; fastMRI [92] MRI Image synthesis MSE; SSIM; PSNR Zero-shot learned adversarial Transformers Huang et al [106] SDAUT 2D Calgary Campinas [107] improving diagnostic accuracy and treatment planning. In recent years, generative adversarial learning (GAN) has been combined with Transformers to carry out tasks such as image generation and reconstruction, thus improving modality transfer capabilities.…”

“…Data collection and annotation difficulties: The collection and annotation of complex biomedical data presents significant challenges in neuroscientific research and clinical practice. While techniques such as data synthesis [93] and harmonization [82] offer potential solutions for enhancing data and adapting across different domains, they can be limited by the inherent complexities and variability of biomedical data. Transfer learning [127] and zero-shot learning, [105] despite their advantages in utilizing existing datasets, often face difficulties due to discrepancies in data distribution and small sample sizes in specialized clinical domains.…”

“…In situations where sequence data is accessible, it can be advantageous for SR tasks. For example, Wang et al [93] introduced SIFormer, a method specifically developed for simultaneous superresolution and missing data imputation in MRI sequences. SIFormer represents a promising advancement in MRI sequence acquisition, which could potentially enhance clinical imaging procedures.…”

Dual contrast attention-guided multi-frequency fusion for multi-contrast MRI super-resolution

“…Deep supervision of feature fusion improves model performance (Qu et al 2022). Therefore, inspired by Lee et al (2015), Wang et al (2023), we incorporated a deeply-supervised mechanism between the HFFD and the LFFD to improve the reconstruction of the texture details and low-frequency information of the target contrast image. The structure of this mechanism is shown in the fourth stage in the gray part of figure 2.…”