Latent space manipulation for high-resolution medical image synthesis via the StyleGAN

Abstract

Introduction: This paper explores the potential of the StyleGAN model as an high-resolution image generator for synthetic medical images. The possibility to generate sample patient images of different modalities can be helpful for training deep learning algorithms as e.g. a data augmentation technique. Methods: The StyleGAN model was trained on Computed Tomography (CT) and T2-weighted Magnetic Resonance (MR) images from 100 patients with pelvic malignancies. The resulting model was investigated with regards to three features: Image Modality, Sex, and Longitudinal Slice Position. Further, the style transfer feature of the StyleGAN was used to move images between the modalities. The root-mean-squard error (RMSE) and the Mean Absolute Error (MAE) were used to quantify errors for MR and CT, respectively. Results: We demonstrate how these features can be transformed by manipulating the latent style vectors, and attempt to quantify how the errors change as we move through the latent style space. The best results were achieved by using the style transfer feature of the StyleGAN (58.7 HU MAE for MR to CT and 0.339 RMSE for CT to MR). Slices below and above an initial central slice can be predicted with an error below 75 HU MAE and 0.3 RMSE within 4 cm for CT and MR, respectively. Discussion: The StyleGAN is a promising model to use for generating synthetic medical images for MR and CT modalities as well as for 3D volumes.

Automated Sewer Defects Detection Using Style-Based Generative Adversarial Networks and Fine-Tuned Well-Known CNN Classifier

“…It is reported to be consistent with increasing disturbances and human judgments in assessing the authenticity and variability of generated images [13,61]. FID calculates the Wasserstein-2 distance between the synthetic images and the real images in the Inception-v3 neural network feature space, see (2) [15,21,62]. A lower FID value indicates a higher image quality and diversity.…”

“…The StyleGAN network is an evolution of the PGGAN where the generator first learns to generate low-resolution images and then progressively produces high-resolution images [15]. It incorporates two improvement strategies to enhance the model performance in terms of image quality, training speed and model stability [13,14,21,27].…”

“…Although these deep learning methods have showed great potential, they in general suffered from major drawbacks of limited training datasets. In fact, in order to avoid model overfitting and produce reliable performance, the need for large and high-quality training datasets has increased dramatically in recent years [13][14][15]. However, most sewer defects datasets are limited and unevenly distributed, which seriously restricts the detection performance of the deep learning methods.…”

“…Stochasticity and style transfer have been incorporated to improve model performance. Furthermore, with aid of transfer learning (TL) [15,22,23], the model training time can be greatly shortened and the image quality can further be improved. However, to the best of our knowledge, previous studies on data augmentation in the field of sewer defects detection were mainly limited to the traditional methods, such as geometric and color transformations [4,22].…”

A review of medical image data augmentation techniques for deep learning applications

“…The main challenges for GAN‐based image synthesis typically are maintaining training stability and generating high quality (clear, high‐resolution) images. To overcome these challenges, many previously proposed GAN variants, with improved network architecture and mathematical optimisation, have been adopted in Ref 53,87,91,131–140. as listed in Table 1.…”