Semantic Image Synthesis via Adversarial Learning

Dong, Hao; Yu, Simiao; Wu, Chao; Guo, Yike

doi:10.1109/iccv.2017.608

Cited by 238 publications

(244 citation statements)

References 17 publications

Supporting

Mentioning

241

Contrasting

Order By: Relevance

“…Additional prior information can be discrete labels, text and images [55], [56]. In this study, a GAN conditioned on images was used and Figure 1 shows the overall framework of our conditional GAN-based CS-MRI architecture.…”

Section: Proposed Methodsmentioning

confidence: 99%

DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction

Yang

Dong

et al. 2018

IEEE Trans. Med. Imaging

Self Cite

853

628

View full text Add to dashboard Cite

Abstract-Compressed Sensing Magnetic Resonance Imaging (CS-MRI) enables fast acquisition, which is highly desirable for numerous clinical applications. This can not only reduce the scanning cost and ease patient burden, but also potentially reduce motion artefacts and the effect of contrast washout, thus yielding better image quality. Different from parallel imaging based fast MRI, which utilises multiple coils to simultaneously receive MR signals, CS-MRI breaks the Nyquist-Shannon sampling barrier to reconstruct MRI images with much less required raw data. This paper provides a deep learning based strategy for reconstruction of CS-MRI, and bridges a substantial gap between conventional non-learning methods working only on data from a single image, and prior knowledge from large training datasets. In particular, a novel conditional Generative Adversarial Networks-based model (DAGAN) is proposed to reconstruct CS-MRI. In our DAGAN architecture, we have designed a refinement learning method to stabilise our U-Net based generator, which provides an endto-end network to reduce aliasing artefacts. To better preserve texture and edges in the reconstruction, we have coupled the adversarial loss with an innovative content loss. In addition, we incorporate frequency domain information to enforce similarity in both the image and frequency domains. We have performed comprehensive comparison studies with both conventional CS-MRI reconstruction methods and newly investigated deep learning approaches. Compared to these methods, our DAGAN method provides superior reconstruction with preserved perceptual image details. Furthermore, each image is reconstructed in about 5 ms, which is suitable for real-time processing.

show abstract

Section: Proposed Methodsmentioning

confidence: 99%

DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction

Yang

Dong

et al. 2018

IEEE Trans. Med. Imaging

Self Cite

853

628

View full text Add to dashboard Cite

show abstract

“…[205,206,207]. These can also be used for biological image synthesis [208,209] and text-to-image synthesis [210,211,212]. 36 Recently, a group of researchers from NVIDIA, MGH & BWH Center for Clinical Data Science in Boston, and the Mayo Clinic in Rochester [213] designed a clever approach to generate synthetic abnormal MRI images with brain tumors by training a GAN based on pix2pix 37 using two publicly available data sets of brain MRI (ADNI and the BRATS'15 Challenge, and later also the Ischemic Stroke Lesion Segmentation ISLES'2018 Challenge).…”

Section: Image Synthesismentioning

confidence: 99%

An overview of deep learning in medical imaging focusing on MRI

Lundervold

2019

Zeitschrift für Medizinische Physik

1,430

883

View full text Add to dashboard Cite

What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI.Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of machine learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging. Artificial neural networksArtificial neural networks (ANNs) is one of the most famous machine learning models, introduced already in the 1950s, and actively studied since [41, Chapter 1.2]. 11 Roughly, a neural network consists of a number of connected computational units, called neurons, arranged in layers. There's an input layer where data enters the network, followed by one or more hidden layers transforming the data as it flows through, before ending at an output layer that produces the neural network's predictions. The network is trained to output useful predictions by identifying patterns in a set of labeled training data, fed through the network while the outputs are compared with the actual labels by an objective function. During training the network's parametersthe strength of each neuron-is tuned until the patterns identified by the network result in good 10 https://www.deeplearningbook.org/ 11 The loose connection between artificial neural networks and neural networks in the brain is often mentioned, but quite over-blown considering the complexity of biological neural networks. However, there is some interesting recent work connecting neuroscience and artificial neural networks, indicating an increase in the cross-fertilization between the two fields [43,44,45].3 predictions for the training data. Once the patterns are learned, the network can be used to make predictions on new, unseen data, i.e. generalize to new data.It h...

show abstract

“…Deep image synthesis The seminal work of pix2pix [16] trains a deep neural network to translate an image from one domain, such as a semantic labeling, into another domain, such as a realistic image, using paired training data. Imageto-image (I2I) translation has since been applied to many tasks [5,24,32,49,47,50]. Several works propose im-provements to stabilize training and allow for high-quality image synthesis [18,46,47].…”

Section: Related Workmentioning

confidence: 99%

Neural Rerendering in the Wild

Meshry

Goldman

Khamis

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

200

141

View full text Add to dashboard Cite

We explore total scene capture -recording, modeling, and rerendering a scene under varying appearance such as season and time of day. Starting from internet photos of a tourist landmark, we apply traditional 3D reconstruction to register the photos and approximate the scene as a point cloud. For each photo, we render the scene points into a deep framebuffer, and train a neural network to learn the mapping of these initial renderings to the actual photos. This rerendering network also takes as input a latent appearance vector and a semantic mask indicating the location of transient objects like pedestrians. The model is evaluated on several datasets of publicly available images spanning a broad range of illumination conditions. We create short videos demonstrating realistic manipulation of the image viewpoint, appearance, and semantic labeling. We also compare results with prior work on scene reconstruction from internet photos.

show abstract

Semantic Image Synthesis via Adversarial Learning

Cited by 238 publications

References 17 publications

DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction

DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction

An overview of deep learning in medical imaging focusing on MRI

Neural Rerendering in the Wild

Contact Info

Product

Resources

About