Cross-Modality Image Synthesis from Unpaired Data Using CycleGAN

Hiasa, Yuta; Otake, Yoshito; Takao, Masaki; Matsuoka, Takumi; Takashima, Kazuma; Carass, Aaron; Prince, Jerry L.; Sugano, Nobuhiko; Sato, Yoshinobu

doi:10.1007/978-3-030-00536-8_4

Cited by 149 publications

(101 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adversarial loss functions have recently been demonstrated for various medical imaging applications with reliable capture of high-frequency texture information [28]- [48]. In the domain of cross-modality image synthesis, important applications include CT to PET synthesis [29], [40], MR to CT synthesis [28], [33], [38], [42], [48], CT to MR synthesis [36], and retinal vessel map to image synthesis [35], [41]. Inspired by this success, here we introduce conditional GAN models for synthesizing images of The pGAN method is based on a conditional adversarial network with a generator G, a pre-trained VGG16 network V, and a discriminator D. Given an input image in a source contrast (e.g., T 1 -weighted), G learns to generate the image of the same anatomy in a target contrast (e.g., T 2 -weighted).…”

Section: Introductionmentioning

confidence: 99%

Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks

Dar

Yurt

Karacan

et al. 2019

IEEE Trans. Med. Imaging

401

281

View full text Add to dashboard Cite

Acquiring images of the same anatomy with multiple different contrasts increases the diversity of diagnostic information available in an MR exam. Yet, the scan time limitations may prohibit the acquisition of certain contrasts, and some contrasts may be corrupted by noise and artifacts. In such cases, the ability to synthesize unacquired or corrupted contrasts can improve diagnostic utility. For multi-contrast synthesis, the current methods learn a nonlinear intensity transformation between the source and target images, either via nonlinear regression or deterministic neural networks. These methods can, in turn, suffer from the loss of structural details in synthesized images. Here, in this paper, we propose a new approach for multi-contrast MRI synthesis based on conditional generative adversarial networks. The proposed approach preserves intermediate-to-high frequency details via an adversarial loss, and it offers enhanced synthesis performance via pixel-wise and perceptual losses for registered multi-contrast images and a cycle-consistency loss for unregistered images. Information from neighboring cross-sections are utilized to further improve synthesis quality. Demonstrations on T 1-and T 2-weighted images from healthy subjects and patients clearly indicate the superior performance of the proposed approach compared to the previous state-of-the-art methods. Our synthesis approach can help improve the quality and versatility Manuscript

show abstract

Section: Introductionmentioning

confidence: 99%

Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks

Dar

Yurt

Karacan

et al. 2019

IEEE Trans. Med. Imaging

401

281

View full text Add to dashboard Cite

show abstract

“…Applying this idea to the field of medical image synthesis, Nie et al enhanced the conditional GAN structure with an auto-context model and achieved good results in the task of tomographic CT image synthesis based on its MR pendant 6 . Furthermore, in 7,[25][26][27] , the successful training of a GAN based on unpaired MR and CT images was shown. As a result of the achieved successes, GANs are now a frequently used tool in medical imaging research that extends beyond the realm of image-to-image translation [28][29][30][31][32] .…”

Section: Image-to-image Translationmentioning

confidence: 99%

Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging

Stimpel

Syben

Würfl

et al. 2019

Sci Rep

View full text Add to dashboard Cite

Hybrid X-ray and magnetic resonance (MR) imaging promises large potential in interventional medical imaging applications due to the broad variety of contrast of MRI combined with fast imaging of X-ray-based modalities. To fully utilize the potential of the vast amount of existing image enhancement techniques, the corresponding information from both modalities must be present in the same domain. For image-guided interventional procedures, X-ray fluoroscopy has proven to be the modality of choice. Synthesizing one modality from another in this case is an ill-posed problem due to ambiguous signal and overlapping structures in projective geometry. To take on these challenges, we present a learning-based solution to MR to X-ray projection-to-projection translation. We propose an image generator network that focuses on high representation capacity in higher resolution layers to allow for accurate synthesis of fine details in the projection images. Additionally, a weighting scheme in the loss computation that favors high-frequency structures is proposed to focus on the important details and contours in projection imaging. The proposed extensions prove valuable in generating X-ray projection images with natural appearance. Our approach achieves a deviation from the ground truth of only 6% and structural similarity measure of 0.913 ± 0.005. In particular the high frequency weighting assists in generating projection images with sharp appearance and reduces erroneously synthesized fine details.

show abstract

“…There are a few observations about GANs: (a) CycleGAN does not guarantee consistent translation of minor anatomical structures and boundaries [53], and thus needs additional constraints like gradient [53] and shape consistency [51]. For instance, Jiang et al, [54] incorporated tumor-shape and feature-based losses to preserve tumors while translating CT data to MRI data; (b) Attention networks can account for varying transferability of different image regions [55].…”

Section: Bidirectional Translationmentioning

confidence: 99%

Advancing Medical Imaging Informatics by Deep Learning-Based Domain Adaptation

Choudhary

Zhu

et al. 2020

Yearb Med Inform

View full text Add to dashboard Cite

Introduction: There has been a rapid development of deep learning (DL) models for medical imaging. However, DL requires a large labeled dataset for training the models. Getting large-scale labeled data remains a challenge, and multi-center datasets suffer from heterogeneity due to patient diversity and varying imaging protocols. Domain adaptation (DA) has been developed to transfer the knowledge from a labeled data domain to a related but unlabeled domain in either image space or feature space. DA is a type of transfer learning (TL) that can improve the performance of models when applied to multiple different datasets. Objective: In this survey, we review the state-of-the-art DL-based DA methods for medical imaging. We aim to summarize recent advances, highlighting the motivation, challenges, and opportunities, and to discuss promising directions for future work in DA for medical imaging. Methods: We surveyed peer-reviewed publications from leading biomedical journals and conferences between 2017-2020, that reported the use of DA in medical imaging applications, grouping them by methodology, image modality, and learning scenarios. Results: We mainly focused on pathology and radiology as application areas. Among various DA approaches, we discussed domain transformation (DT) and latent feature-space transformation (LFST). We highlighted the role of unsupervised DA in image segmentation and described opportunities for future development. Conclusion: DA has emerged as a promising solution to deal with the lack of annotated training data. Using adversarial techniques, unsupervised DA has achieved good performance, especially for segmentation tasks. Opportunities include domain transferability, multi-modal DA, and applications that benefit from synthetic data.

show abstract

Cross-Modality Image Synthesis from Unpaired Data Using CycleGAN

Cited by 149 publications

References 16 publications

Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks

Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial Networks

Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging

Advancing Medical Imaging Informatics by Deep Learning-Based Domain Adaptation

Contact Info

Product

Resources

About