RS-Net: Regression-Segmentation 3D CNN for Synthesis of Full Resolution Missing Brain MRI in the Presence of Tumours

Abstract

Accurate synthesis of a full 3D MR image containing tumours from available MRI (e.g. to replace an image that is currently unavailable or corrupted) would provide a clinician as well as downstream inference methods with important complementary information for disease analysis. In this paper, we present an end-to-end 3D convolution neural network that takes a set of acquired MR image sequences (e.g. T1, T2, T1ce) as input and concurrently performs (1) regression of the missing full resolution 3D MRI (e.g. FLAIR) and (2) segmentation of the tumour into subtypes (e.g. enhancement, core). The hypothesis is that this would focus the network to perform accurate synthesis in the area of the tumour. Experiments on the BraTS 2015 and 2017 datasets [1] show that: (1) the proposed method gives better performance than state-of-the art methods in terms of established global evaluation metrics (e.g. PSNR), (2) replacing real MR volumes with the synthesized MRI does not lead to significant degradation in tumour and sub-structure segmentation accuracy. The system further provides uncertainty estimates based on Monte Carlo (MC) dropout [11] for the synthesized volume at each voxel, permitting quantification of the system's confidence in the output at each location.

Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network

“…Though all the methods discussed above propose a multiinput method, none of the methods have been proposed to synthesize multiple missing sequences (multi-output), and in one single pass. All three methods [16], [5], and [24] synthesize only one sequence (either T 2f lair or T 2 , many-to-one setting) in the presence of varying number of input sequences, while [23] only synthesizes MRA using information from multiple inputs (many-to-one). Although the work presented in [23] is close to our proposed method, theirs is not a truly multimodal network (many-to-many), since there is no empirical evidence that their method will generalize to multiple scenarios.…”

Multimodal MRI synthesis using unified generative adversarial networks

“…As the rapid growth of applying deep learning in MRI, [19][20][21] recently, deep learning-based end-to-end frameworks have been investigated for multimodal MR image synthesis. 13,[22][23][24][25][26][27][28][29][30][31][32] Especially, the achievable accuracy of synthesis has been highly improved with the superior image synthesis capability of generative adversarial networks (GANs). 33 These deep neural network-based methods can be grouped into three main categories depending on their input/output modalities: (a) single-input single-output (SISO), (b) multi-input singleoutput (MISO), (c) multi-input multi-output (MIMO).…”

“…A regression-segmentation 3D convolutional neural network was implemented to simultaneously synthesize missing MRI modality and segment tumor regions. 22 A scalable GANbased model was developed to flexibly take arbitrary subsets of the multiple modalities as input and generate the target modality. 27 Very recently, Zhou et al proposed a hybrid-fusion network consisting of modality-specific, multi-modal fusion, and image synthesis subnetworks to learn the correlations among multiple modalities with enhanced multi-level fusion strategy thus improving the performance of synthesis.…”

Prior-Guided Image Reconstruction for Accelerated Multi-Contrast MRI via Generative Adversarial Networks

“…A fundamentally different approach for accelerated MRI is to perform fully-sampled acquisitions of a subset of the desired contrasts (i.e., source contrasts), and then to synthesize missing contrasts (i.e., target contrasts). This approach requires an intensity-based mapping model estimated using a collection of image pairs in both source and target contrast [23]- [50]. The model can be based on sparse linear mapping between source and target patches [32], or deep neural networks for enhanced accuracy [28], [34]- [36], [39]- [50].…”

“…This approach requires an intensity-based mapping model estimated using a collection of image pairs in both source and target contrast [23]- [50]. The model can be based on sparse linear mapping between source and target patches [32], or deep neural networks for enhanced accuracy [28], [34]- [36], [39]- [50]. Although deep models for synthesis are promising, local inaccuracies may occur in synthesized images when the source contrast is less sensitive to differences in relaxation parameters of two tissues compared to the target contrast, or vice versa.…”