Deep Image Translation for Enhancing Simulated Ultrasound Images

Abstract

Ultrasound simulation based on ray tracing enables the synthesis of highly realistic images. It can provide an interactive environment for training sonographers as an educational tool. However, due to high computational demand, there is a trade-off between image quality and interactivity, potentially leading to sub-optimal results at interactive rates. In this work we introduce a deep learning approach based on adversarial training that mitigates this trade-off by improving the quality of simulated images with constant computation time. An image-to-image translation framework is utilized to translate low quality images into high quality versions. To incorporate anatomical information potentially lost in low quality images, we additionally provide segmentation maps to image translation. Furthermore, we propose to leverage information from acoustic attenuation maps to better preserve acoustic shadows and directional artifacts, an invaluable feature for ultrasound image interpretation. The proposed method yields an improvement of 7.2% in Fréchet Inception Distance and 8.9% in patch-based Kullback-Leibler divergence.

Learning ultrasound rendering from cross-sectional model slices for simulated training

“…Aggregating such maps along ultrasound propagation then yields the so-called integral attenuation maps A are derived for each frame in real-time. b Network architecture of the generator used in [28]. c Network architecture proposed herein to address the challenges of image translation from segmentation maps to complex Bmode image content.…”

“…Therefore, acoustic shadow is a cumulative effect which requires a large network receptive field and significant network modeling ability. The authors in [28] propose to provide this cumulative information in a form of an integral attenuation map and they demonstrates that providing this additional information to the network greatly helps to reproduce acoustic shadows. Integral attenuation maps are computed as a = e − z i=0 μ[i] with the image depth z and frequency-dependent and tissue specific attenuation coefficient μ.…”

“…A constrained cycleGAN is proposed in [9] for unpaired translation from echocardiographic images acquired with point-of-care ultrasound to high-end devices. Recently, a GAN-based image translation framework has been proposed in [28] for recovering high quality US images equivalent to computationally expensive ray-based simulations using low-quality and thus faster simulations, by also leveraging information from corresponding segmentation and integral attenuation maps. However, rendering such low-quality ultrasound images still requires additional computation time and sophisticated hardware resources.…”

“…Herein we adopt a Monte-carlo simulation presented in [15] for simulating the same abdominal scene as in previous work [28]. Since low-quality images as used in [28] provide information about wave interactions and speckle textures, omitting this modality in the network input renders the translation task much more challenging. We carefully inspect the generator architecture and propose several crucial architectural improvements to enable realistic quality images from merely the segmentation maps as input.…”

“…With the above mentioned architectural changes in contrast to [28], our proposed network is able to simulate US images with realistic structural and textural content. To the best of our knowledge, this is the first work investigating deep neural networks to generate simulated ultrasound images using only structural tissue cross-sections.…”