Denoising diffusion probabilistic models for 3D medical image generation

Khader, Firas; Müller‐Franzes, Gustav; Arasteh, Soroosh Tayebi; Han, Tianyu; Haarburger, Christoph; Schulze-Hagen, Maximilian; Schad, Philipp; Engelhardt, Sandy; Baeßler, Bettina; Foersch, Sebastian; Stegmaier, Johannes; Kühl, Christiane; Nebelung, Sven; Kather, Jakob Nikolas; Truhn, Daniel

doi:10.1038/s41598-023-34341-2

Cited by 43 publications

(18 citation statements)

References 29 publications

(15 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For our experiments, we chose six recent 3D models with a range of sizes, from 1.8 million (M) to 27M parameters. This included: the SwinT vision transformer with 27M parameters, as adapted for video analysis [13], the multi-instance neuroimage transformer (MiNiT) with 3.6M parameters, and the Neuroimage Transformer (NiT) with 2M parameters, recently adapted for neuroimage applications [20]. Further, we used a commonly used architecture in neuroimaging -the DenseNet121 convolution-al neural network (CNN) [9] with 11M parameters, and a scaled down version with 10 times fewer parameters, Tiny-DenseNet CNN [6] with 1.8M parameters.…”

Section: Vision Architecturesmentioning

confidence: 99%

Video and synthetic MRI pre-training of 3D vision architectures for neuroimage analysis

Dhinagar,

Singh,

Ozarkar

et al. 2024

Medical Imaging 2024: Computer-Aided Diagnosis

View full text Add to dashboard Cite

Transfer learning represents a recent paradigm shift in the way we build artificial intelligence (AI) systems. In contrast to training task-specific models, transfer learning involves pre-training deep learning models on a large corpus of data and minimally fine-tuning them for adaptation to specific tasks. Even so, for 3D medical imaging tasks, we do not know if it is best to pre-train models on natural images, medical images, or even synthetically generated MRI scans or video data. To evaluate these alternatives, here we benchmarked vision transformers (ViTs) and convolutional neural networks (CNNs), initialized with varied upstream pre-training approaches. These methods were then adapted to three unique downstream neuroimaging tasks with a range of difficulty: Alzheimer's disease (AD) and Parkinson's disease (PD) classification, "brain age" prediction. Experimental tests led to the following key observations: 1. Pre-training improved performance across all tasks including a boost of 7.5% for AD classification and 4.5% for PD classification for the ViT and 19.1% for PD classification and reduction in brain age prediction error by 1.26 years for CNNs, 2. Pre-training on large-scale video or synthetic MRI data boosted performance of ViTs, 3. CNNs were robust in limited-data settings, and in-domain pretraining enhanced their performances, 4. Pre-training improved generalization to out-of-distribution datasets and sites. Overall, we benchmarked different vision architectures, revealing the impact of pre-training them with emerging datasets for model initialization. The resulting pre-trained models can be adapted to a range of downstream neuroimaging tasks, especially when training data for a domain-specific target task is limited.

show abstract

Section: Vision Architecturesmentioning

confidence: 99%

Video and synthetic MRI pre-training of 3D vision architectures for neuroimage analysis

Dhinagar,

Singh,

Ozarkar

et al. 2024

Medical Imaging 2024: Computer-Aided Diagnosis

View full text Add to dashboard Cite

show abstract

“…[23][24][25]29 Diffusion models have been used to generate magnetic resonance imaging (MRI) and computed tomography (CT) scans to augment the training datasets of deep learning models. 21,27,30 Despite advances in generative models, limitations persist in achieving synthetic biological images that look realistic, as assessed by rigorous metrics. [21][22][23][24][25][26][27][28][31][32][33] Issues such as the accurate representation of subtle or rare textures, cell arrangements, and tissue boundaries are areas of active research.…”

Section: Introductionmentioning

confidence: 99%

“…21,27,30 Despite advances in generative models, limitations persist in achieving synthetic biological images that look realistic, as assessed by rigorous metrics. [21][22][23][24][25][26][27][28][31][32][33] Issues such as the accurate representation of subtle or rare textures, cell arrangements, and tissue boundaries are areas of active research. 22,26 Here, we explore interpolation techniques, such as frame interpolation for large motion (FILM), to enhance the resolution of 3D biological images.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Generative interpolation and restoration of images using deep learning for improved 3D tissue mapping

Joshi,

Forjaz,

Sang Han

et al. 2024

Preprint

View full text Add to dashboard Cite

The development of novel imaging platforms has improved our ability to collect and analyze large three-dimensional (3D) biological imaging datasets. Advances in computing have led to an ability to extract complex spatial information from these data, such as the composition, morphology, and interactions of multi-cellular structures, rare events, and integration of multi-modal features combining anatomical, molecular, and transcriptomic (among other) information. Yet, the accuracy of these quantitative results is intrinsically limited by the quality of the input images, which can contain missing or damaged regions, or can be of poor resolution due to mechanical, temporal, or financial constraints. In applications ranging from intact imaging (e.g. light-sheet microscopy and magnetic resonance imaging) to sectioning based platforms (e.g. serial histology and serial cryo-electron microscopy), the quality and resolution of imaging data has become paramount. Here, we address these challenges by leveraging frame interpolation for large image motion (FILM), a generative AI model originally developed for temporal interpolation, for spatial interpolation of a range of 3D image types. Comparative analysis demonstrates the superiority of FILM over traditional linear interpolation to produce functional synthetic images, due to its ability to better preserve biological information including microanatomical features and cell counts, as well as image quality, such as contrast, variance, and luminance. FILM repairs tissue damages in images and reduces stitching artifacts. We show that FILM can decrease imaging time by synthesizing skipped images. We demonstrate the versatility of our method with a wide range of imaging modalities (histology, tissue-clearing/light-sheet microscopy, magnetic resonance imaging, cryo-electron microscopy), species (human, mouse), healthy and diseased tissues (pancreas, lung, brain), staining techniques (IHC, H&E), and pixel resolutions (8 nm, 2 μm, 1mm). Overall, we demonstrate the marked potential of generative AI in improving the resolution, throughput, and quality of biological image datasets, enabling improved 3D imaging.

show abstract

“…While 2D- or 3D-patch-based approaches showcase significant utility for selected downstream tasks, particular downstream tasks such as hemodynamic modeling (Rundfeldt et al, 2024) or 3D classification demand realistic anatomical accuracy and spatial continuity from the synthesized vessel volumes. To tackle the research gap of 3D modeling, prior works have proposed 3D adaptations of generative adversarial networks (Hong et al, 2021; Mensing et al, 2022) and denoising diffusion probabilistic models for medical images (Khader et al, 2023).…”

Section: Introductionmentioning

confidence: 99%

Generative Modeling of the Circle of Willis Using 3D-StyleGAN

Aydin,

Hilbert,

Koch

et al. 2024

Preprint

View full text Add to dashboard Cite

The circle of Willis (CoW) is a network of cerebral arteries with significant inter-individual anatomical variations. Deep learning has been used to characterize and quantify the status of the CoW in various applications for the diagnosis and treatment of cerebrovascular disease. In medical imaging, the performance of deep learning models is limited by the diversity and size of training datasets. To address medical data scarcity, generative adversarial networks (GANs) have been applied to generate synthetic vessel neuroimaging data. However, the proposed methods produce synthetic data with limited anatomical fidelity or downstream utility in tasks concerning vessel characteristics.We adapted the StyleGANv2 architecture to 3D to synthesize Time-of-Flight Magnetic Resonance Angiography (TOF MRA) volumes of the CoW. For generative modeling, we used 1782 individual TOF MRA scans from 6 open source datasets. To train the adapted 3D StyleGAN model with limited data we employed differentiable data augmentations and used mixed precision and a cropped region of interest of size 32×128×128 to tackle computational constraints. The performance was evaluated quantitatively using the Fréchet Inception Distance (FID), MedicalNet distance (MD) and Area Under the Curve of the Precision and Recall Curve for Distributions (AUC-PRD). Qualitative analysis was performed via a visual Turing test. We demonstrated the utility of generated data in a downstream task of multiclass semantic segmentation of CoW arteries. Vessel segmentation performance was assessed quantitatively using the Dice coefficient and the Hausdorff distance.The best-performing 3D StyleGANv2 architecture generated high-quality and diverse synthetic TOF MRA volumes (FID: 12.17, MD: 0.00078, AUC-PRD: 0.9610). Multiclass vessel segmentation models trained on synthetic data alone achieved comparable performance to models trained using real data in most arteries.In conclusion, generative modeling of the Circle of Willis via synthesis of 3D TOF MRA data paves the way for generalizable deep learning applications in cerebrovascular disease. In the future, the extensions of the provided methodology to other medical imaging problems or modalities with the inclusion of pathological datasets has the potential to advance the development of more robust models for clinical applications.

show abstract

Denoising diffusion probabilistic models for 3D medical image generation

Cited by 43 publications

References 29 publications

Video and synthetic MRI pre-training of 3D vision architectures for neuroimage analysis

Video and synthetic MRI pre-training of 3D vision architectures for neuroimage analysis

Generative interpolation and restoration of images using deep learning for improved 3D tissue mapping

Generative Modeling of the Circle of Willis Using 3D-StyleGAN

Contact Info

Product

Resources

About