Simulating patho-realistic ultrasound images using deep generative networks with adversarial learning

Abstract

Ultrasound imaging makes use of backscattering of waves during their interaction with scatterers present in biological tissues. Simulation of synthetic ultrasound images is a challenging problem on account of inability to accurately model various factors of which some include intra-/inter scanline interference, transducer to surface coupling, artifacts on transducer elements, inhomogeneous shadowing and nonlinear attenuation. Current approaches typically solve wave space equations making them computationally expensive and slow to operate. We propose a generative adversarial network (GAN) inspired approach for fast simulation of patho-realistic ultrasound images. We apply the framework to intravascular ultrasound (IVUS) simulation. A stage 0 simulation performed using pseudo B-mode ultrasound image simulator yields speckle mapping of a digitally defined phantom. The stage I GAN subsequently refines them to preserve tissue specific speckle intensities. The stage II GAN further refines them to generate high resolution images with patho-realistic speckle profiles. We evaluate patho-realism of simulated images with a visual Turing test indicating an equivocal confusion in discriminating simulated from real. We also quantify the shift in tissue specific intensity distributions of the real and simulated images to prove their similarity.Index Terms-Adversarial learning, deep convolutional neural network, generative adversarial network, intravascular ultrasound, ultrasound simulation.

A neural lens for super-resolution biological imaging

“…While parametric modelling approaches have been applied in order to enhance medical imaging [8], recent work using deep learning approaches has also shown the potential for the application of neural networks (NNs) for enhancing image resolution. Examples for such an application include, a 4x upscaling on photographic images [9], optical microscopy (improving the resolution from 40x to 100x) [10], dental imaging [11], phase imaging [12], fluorescence microscopy [13], magnetic resonance imaging [14], SEM imaging [15,16], positronemission tomography [17], stochastic optical reconstruction microscopy [18], and ultrasound imaging [19]. NNs are also well-suited to the classification of objects in images, and accordingly the classification of biological, pollution and colloidal particles from images and scattering patterns has also been demonstration [20][21][22][23][24].…”

Hi-Net: Hybrid-Fusion Network for Multi-Modal MR Image Synthesis

“…Li et al [33] used a deep learning model to estimate the missing positron emission tomography (PET) data from the corresponding MRI data. In addition, GAN-based methods have also achieved promising results in synthesizing various types of medical images, such as CT images [15], retinal images [34], [35], MR images [19], ultrasound images [36], [37] and so on. For example, Nie et al [15] utilized MR images to synthesize computed tomography (CT) images with a context-aware GAN model.…”

Deep Learning in Cardiology

“…Method Application/Notes a CT Lessman 2016 [195] CNN detect coronary calcium using three independently trained CNNs Shadmi 2018 [196] DenseNet compared DenseNet and u-net for detecting coronary calcium Cano 2018 [197] CNN 3D regression CNN for calculation of the Agatston score Wolterink 2016 [198] CNN detect coronary calcium using three CNNs for localization and two CNNs for detection Santini 2017 [199] CNN coronary calcium detection using a seven layer CNN on image patches Lopez 2017 [200] CNN thrombus volume characterization using a 2D CNN and postprocessing Hong 2016 [201] DBN detection, segmentation, classification of abdominal aortic aneurysm using DBN and image patches Liu 2017 [202] CNN left atrium segmentation using a twelve layer CNN and active shape model (STA13) de Vos 2016 [203] CNN 3D localization of anatomical structures using three CNNs, one for each orthogonal plane Moradi 2016 [204] CNN detection of position for a given CT slice using a pretrained VGGnet, handcrafted features and SVM Zheng 2015 [205] Multiple carotid artery bifurcation detection using multi-layer perceptrons and probabilistic boosting-tree Montoya 2018 [206] ResNet 3D reconstruction of cerebral angiogram using a 30 layer ResNet Zreik 2018 [207] CNN, AE identify coronary artery stenosis using CNN for LV segmentation and an AE, SVM for classification Commandeur 2018 [208] CNN quantification of epicardial and thoracic adipose tissue from non-contrast CT Gulsun 2016 [209] CNN extract coronary centerline using optimal path from computed flow field and a CNN for refinement CNN carotid intima media thickness video interpretation using two CNNs with two layers on Ultrasound Tom 2017 [226] GAN IVUS image generation using two GANs (IV11) Wang 2017 [227] CNN breast arterial calcification using a ten layer CNN on mammograms Liu 2017 [228] CNN CAC detection using CNNs on 1768 X-Rays Pavoni 2017 [229] CNN denoising of percutaneous transluminal coronary angioplasty images using four layer CNN Nirschl 2018 [230] CNN trained a patch-based six layer CNN for identifying heart failure in endomyocardial biopsy images Betancur 2018 [231] CNN trained a three layer CNN for obstructive CAD prediction from myocardial perfusion imaging a Results from these imaging modalities are not reported in this review because they were highly variable in terms of the research question they were trying to solve and highly inconsistent in respect with the use of metrics. Additionally all papers use private databases besides…”

“…This model performs much better than a previous handcrafted method by the same authors, which they justify in CNN's capability to learn the appearance of QRS and ROI instead of relying on a thresholded R-peak amplitude and curvature. Tom et al [226] created a Generative Adversarial Network (GAN) based method for fast simulation of realistic IVUS. Stage 0 simulation was performed using pseudo B-mode IVUS simulator and yielded speckle mapping of a digitally defined phantom.…”

“…Both networks try to optimize a loss in a zero-sum game resulting in a generator that produces realistic images. GANs have only been used for simulating patho-realistic IVUS images [226] and the cardiology field has a lot to gain from using this kind of models, especially in the absense of high quality annotated data.…”