Reconstruction of multicontrast MR images through deep learning

Abstract

Purpose Magnetic resonance (MR) imaging with a long scan time can lead to degraded images due to patient motion, patient discomfort, and increased costs. For these reasons, the role of rapid MR imaging is important. In this study, we propose the joint reconstruction of multicontrast brain MR images from down‐sampled data to accelerate the data acquisition process using a novel deep‐learning network. Methods Twenty‐one healthy volunteers (female/male = 7/14, age = 26 ± 4 yr, range 22–35 yr) and 16 postoperative patients (female/male = 7/9, age = 49 ± 9 yr, range 37–62 yr) were scanned on a 3T whole‐body scanner for prospective and retrospective studies, respectively, using both T1‐weighted spin‐echo (SE) and T2‐weighted fast spin‐echo (FSE) sequences. We proposed a network which we term “X‐net” to reconstruct both T1‐ and T2‐weighted images from down‐sampled images as well as a network termed “Y‐net” which reconstructs T2‐weighted images from highly down‐sampled T2‐weighted images and fully sampled T1‐weighted images. Both X‐net and Y‐net are composed of two concatenated subnetworks. We investigate optimal sampling patterns, the optimal patch size for augmentation, and the optimal acceleration factors for network training. An additional Y‐net combined with a generative adversarial network (GAN) was also implemented and tested to investigate the effects of the GAN on the Y‐net performance. Single‐ and joint‐reconstruction parallel‐imaging and compressed‐sensing algorithms along with a conventional U‐net were also tested and compared with the proposed networks. For this comparison, the structural similarity (SSIM), normalized mean square error (NMSE), and Fréchet inception distance (FID) were calculated between the outputs of the networks and fully sampled images. The statistical significance of the performance was evaluated by assessing the interclass correlation and in paired t‐tests. Results The outputs from the two concatenated subnetworks were closer to the fully sampled images compared to those from one subnetwork, with this result showing statistical significance. Uniform down‐sampling led to a statically significant improvement in the image quality compared to random or central down‐sampling patterns. In addition, the proposed networks provided higher SSIM and NMSE values than U‐net, compressed‐sensing, and parallel‐imaging algorithms, all at statistically significant levels. The GAN‐based Y‐net showed a better FID and more realistic images compared to a non‐GAN‐based Y‐net. The performance capabilities of the networks were similar between normal subjects and patients. Conclusions The proposed X‐net and Y‐net effectively reconstructed full images from down‐sampled images, outperforming the conventional parallel‐imaging, compressed‐sensing and U‐net methods and providing more realistic images in combination with a GAN. The developed networks potentially enable us to accelerate multicontrast anatomical MR imaging in routine clinical studies including T1‐and T2‐weighted imaging.

Enhancing magnetic resonance imaging-driven Alzheimer’s disease classification performance using generative adversarial learning

“…This was achieved by allowing the gradient calculated from the FCN classification loss to propagate back to the generator to implicitly convey the disease-related information to the generator. As a result, the classification loss that was propagated provided a momentum for the generator to generate images that contributed to 16 [6,20] 16 [6,20] 16 [4,20] 16 [7,20] 16 [6,20] 14 [7,20] [24,30] 23.5 [18,28] 30 [26,30] 26 [24,30] 23 [20,27] 29 [20,30] 22 [0,30] 29 [25,30] 18 [6,22] Three independent datasets including (a) the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, (b) the Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing (AIBL), and (c) the National Alzheimer's Coordinating Center (NACC) were used for this study lower cross-entropy loss and thus facilitated better image classification. The generator of the GAN model consists of three 3D convolutional blocks in which each convolutional operation was followed by batch normalization and rectified linear unit (ReLu) activation.…”

“…Since its introduction, there has been a surge of interest in the application of GAN frameworks related to the brain. Some of the applications include image generation with improved properties such as achieving super resolution or better quality [6][7][8][9][10][11], data augmentation [12][13][14], segmentation [9,[13][14][15][16], image reconstruction [17][18][19][20], image-to-image translation [21][22][23][24], and motion correction [25,26]. While these important studies have demonstrated the exciting prospect of using GAN architectures, there is a limited amount of work that has focused on utilizing the generated images for subsequent tasks such as disease classification [27].…”

Multimodal MRI synthesis using unified generative adversarial networks

“…As the rapid growth of applying deep learning in MRI, [19][20][21] recently, deep learning-based end-to-end frameworks have been investigated for multimodal MR image synthesis. 13,[22][23][24][25][26][27][28][29][30][31][32] Especially, the achievable accuracy of synthesis has been highly improved with the superior image synthesis capability of generative adversarial networks (GANs). 33 These deep neural network-based methods can be grouped into three main categories depending on their input/output modalities: (a) single-input single-output (SISO), (b) multi-input singleoutput (MISO), (c) multi-input multi-output (MIMO).…”

Deep learning for brain disorders: from data processing to disease treatment

“…The majority of the DL methods operate at the image-level [21][22][23][24][25][26][27][28][29][30][31][32][33][34][35] with the networks learning a mapping from aliased images, reconstructed from under-sampled k-spaces using the inverse Fourier transform, to de-aliased images. The de-aliased image can be the final product [23,25,27,24,31,32,35], be used as input of classic approaches [21] or be combined with the initial under-sampled k-space before a refinement step [26,28,34]. The network architecture can also be designed to mimic a classic iterative reconstruction where steps of reconstruction and DL-based regularization alternate [22,29,30,33].…”