Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging

Abstract

Since the advent of deep convolutional neural networks (DNNs), computer vision has seen an extremely rapid progress that has led to huge advances in medical imaging. Every year, many new methods are reported of in conferences such as the International Conference on Medical Image Computing and Computer Assisted Intervention (MIC-CAI) and Machine Learning for Medical Image Reconstruction (MLMIR), or published online at the preprint server arXiv. There is a plethora of surveys on applications of neural networks in medical imaging (see [1] for a relatively recent comprehensive survey). This article does not aim to cover all aspects of the field but focuses on a particular topic, image-to-image translation. Although the topic may not sound familiar, it turns out that many seemingly irrelevant applications can be understood as instances of image-to-image translation. Such applications include (1) noise reduction, (2) super-resolution, (3) image synthesis, and (4) reconstruction. The same underlying principles and algorithms work for various tasks. Our aim is to introduce some of the key ideas on this topic from a uniform point of view. We introduce core ideas and jargon that are specific to image processing by use of DNNs. Having an intuitive grasp of the core ideas of and a knowledge of technical terms would be of great help to the reader for understanding the existing and future applications.Most of the recent applications which build on imageto-image translation are based on one of two fundamental The two authors contributed equally to this work.

Computed Tomography 3D Super-Resolution with Generative Adversarial Neural Networks: Implications on Unsaturated and Two-Phase Fluid Flow

“…Up to now, these super-resolution methods have not often received attention for geological applications and were mainly addressed by Wang et al [82,83,85] and Wu et al [75] who attempted classical super-resolution methods like neighbour embedding [75,82,[156][157][158] and sparse representation [83,159], although Wang et al [84,85] were also the first to use convolutional neural networks to address the super-resolution problem for geological materials. Compared to these networks, the presented research used deeper neural networks, an encoder-decoder architecture with residual connections and generative training to better conform to approaches in recent super-resolution literature [93,96,100]. Additionally, Shams et al [88] were the first to use a GAN to solve the super-resolution problem and worked directly on segmented images, of which the low resolution images were artificially prepared.…”

WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting

“…From a ML perspective, state‐to‐state weather prediction is similar to image‐to‐image translation. For this sort of problem many deep learning techniques have been developed in recent years (Kaji & Kida, 2019). However, forecasting weather differs in some important ways from typical image‐to‐image applications and raises several open questions.…”

“…veloped in recent years (Kaji & Kida, 2019 olis effect, the deflection of wind caused by the rotation of Earth, which also depends on latitude. A possible solution in the horizontal could be to use spherical convolutions (Cohen et al, 2018;Perraudin et al, 2019;Jiang et al, 2019) or to feed in latitude information to the network.…”

A bird’s-eye view of deep learning in bioimage analysis

“…But even experts are known to suffer from subjectivity, variability, and fatigue. These impediments can potentially be overcome by computational methods, and deep learning in particular has emerged as a key enabling technology for this purpose, as attested by many recent overview articles in the field [95] , [96] , [97] , [98] , [99] , [100] , [101] , [102] .…”