MRI super‐resolution reconstruction for MRI‐guided adaptive radiotherapy using cascaded deep learning: In the presence of limited training data and unknown translation model

Abstract

Purpose Deep learning (DL)‐based super‐resolution (SR) reconstruction for magnetic resonance imaging (MRI) has recently been receiving attention due to the significant improvement in spatial resolution compared to conventional SR techniques. Challenges hindering the widespread implementation of these approaches remain, however. Low‐resolution (LR) MRIs captured in the clinic exhibit complex tissue structures obfuscated by noise that are difficult for a simple DL framework to handle. Moreover, training a robust network for a SR task requires abundant, perfectly matched pairs of LR and high‐resolution (HR) images that are often unavailable or difficult to collect. The purpose of this study is to develop a novel SR technique for MRI based on the concept of cascaded DL that allows for the reconstruction of high‐quality SR images in the presence of insufficient training data, an unknown translation model, and noise. Methods The proposed framework, based on the concept named cascaded deep learning, consists of three components: (a) a denoising autoencoder (DAE) trained using clinical LR noisy MRI scans that have been processed with a nonlocal means filter that generates denoised LR data; (b) a down‐sampling network (DSN) trained with a small amount of paired LR/HR data from volunteers that allows for the generation of perfectly paired LR/HR data for the training of a generative model; and (c) the proposed SR generative model (p‐SRG) trained with data generated by the DSN that maps from LR inputs to HR outputs. After training, LR clinical images may be fed through the DAE and p‐SRG to yield SR reconstructions of the LR input. The application of this framework was explored in two settings: 3D breath‐hold MRI axial SR reconstruction from LR axial scans (<3 sec/vol) and in the enhancement of the spatial resolution of LR 4D‐MRI acquisitions (0.5 sec/vol). Results The DSN produces LR scans from HR inputs with a higher fidelity to true, LR clinical scans compared to conventional k‐space down‐sampling methods based on the metrics of root mean square error (RMSE) and structural similarity index (SSIM). Furthermore, HR outputs generated by the p‐SRG exhibit improved scores in the peak signal‐to‐noise ratio, normalized RMSE, SSIM, and in the blind/reference‐less image spatial quality evaluator assessment compared to conventional approaches to MRI SR. Conclusions The robust, SR reconstruction method for MRI based on the novel cascaded deep learning framework is an end‐to‐end method for producing detail‐preserving SR reconstructions from noisy, LR clinical MRI scans. Fourfold enhancements in spatial resolution facilitate target delineation and motion management during radiation therapy, enabling precise MRI‐guided radiation therapy with 3D LR breath‐hold MRI and 4D‐MRI in a clinically feasible time frame.

Technical Note: Real‐time 3D MRI in the presence of motion for MRI‐guided radiotherapy: 3D Dynamic keyhole imaging with super‐resolution

“…SR images can be generated without transformation models since DL‐based methods utilize direct mapping based on information extracted from previous datasets. Recent studies have shown that the in‐plane resolution of physically scanned low‐spatial resolution 3D MRI can be improved to a four times higher resolution without compromising the image quality …”

“…28,29 As post-processing means, they can generate images with high-SNR and high-spatial resolution within a reasonable processing time frame from low-SNR and low-spatial images obtained with high frame rates. 29 Utilizing super-resolution can overcome a spatial resolution limitation in 3D dynamic keyhole imaging. The aim of this paper is to propose high spatial and temporal resolution 3D MRIs in the presence of motion for real-time 3D MRgRT by combining 3D dynamic keyhole imaging with super-resolution generative (SRG) model.…”

“…Recent studies have shown that the in-plane resolution of physically scanned low-spatial resolution 3D MRI can be improved to a four times higher resolution without compromising the image quality. 29 A cascaded DL-based SRG model used a splitting SR generation process to make a SR model robust compared to the conventional DL-based methods. The cascaded DL-based SR consisted of three training stages: (a) training of a denoising autoencoder (DAE) using noise reduced low-spatial resolution data; (b) training of a downsampling network (DSN) using a paired low-spatial resolution (LR)/high-spatial resolution (HR) data that allows the generation of perfectly paired LR/HR data from the clinically abundant HR scans; and (c) the training of a SRG trained with data generated by the DSN that maps from LR inputs to HR outputs.…”

Deep‐Learning Super‐Resolution MRI: Getting Something From Nothing

“…Image SR, the idea of transforming low‐resolution into high‐resolution images and accurately recovering detail from sparse information, is perhaps a bit of a holy grail of image reconstruction, sounding like something straight out of science fiction, but it is in fact a field with a long history, albeit a more nascent history in Deep Learning. This technique is being applied in many other contexts, from improving the image quality of heart coronary MRAs, to adaptive MRI radiotherapy, to improving the appearance of multiple sclerosis white matter lesions on FLAIR images . However, unlike more generic methods like classical interpolation, the improved performance of many of these deep‐learning SR methods may be bound to a specific examination, protocol, or clinical application.…”

“…This technique is being applied in many other contexts, from improving the image quality of heart coronary MRAs, to adaptive MRI radiotherapy, to improving the appearance of multiple sclerosis white matter lesions on FLAIR images. [3][4][5] However, unlike more generic methods like classical interpolation, the improved performance of many of these deep-learning SR methods may be bound to a specific examination, protocol, or clinical application. Because of the reduced model transparency of such deep-learning methods, continuous realworld evaluation is required, backed by prospective trials evaluating reconstruction from original low-resolution images at the scanner, to establish a safe and reliable evidence base for usage.…”