Semantic Image Synthesis via Adversarial Learning

Dong, Hao; Yu, Simiao; Wu, Chao; Guo, Yike

doi:10.48550/arxiv.1707.06873

Cited by 12 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[205,206,207]. These can also be used for biological image synthesis [208,209] and text-to-image synthesis [210,211,212]. 36 Recently, a group of researchers from NVIDIA, MGH & BWH Center for Clinical Data Science in Boston, and the Mayo Clinic in Rochester [213] designed a clever approach to generate synthetic abnormal MRI images with brain tumors by training a GAN based on pix2pix 37 using two publicly available data sets of brain MRI (ADNI and the BRATS'15 Challenge, and later also the Ischemic Stroke Lesion Segmentation ISLES'2018 Challenge).…”

Section: Image Synthesismentioning

confidence: 99%

An overview of deep learning in medical imaging focusing on MRI

Lundervold

2019

Zeitschrift für Medizinische Physik

1,518

883

View full text Add to dashboard Cite

What has happened in machine learning lately, and what does it mean for the future of medical image analysis? Machine learning has witnessed a tremendous amount of attention over the last few years. The current boom started around 2009 when so-called deep artificial neural networks began outperforming other established models on a number of important benchmarks. Deep neural networks are now the state-of-the-art machine learning models across a variety of areas, from image analysis to natural language processing, and widely deployed in academia and industry. These developments have a huge potential for medical imaging technology, medical data analysis, medical diagnostics and healthcare in general, slowly being realized. We provide a short overview of recent advances and some associated challenges in machine learning applied to medical image processing and image analysis. As this has become a very broad and fast expanding field we will not survey the entire landscape of applications, but put particular focus on deep learning in MRI.Our aim is threefold: (i) give a brief introduction to deep learning with pointers to core references; (ii) indicate how deep learning has been applied to the entire MRI processing chain, from acquisition to image retrieval, from segmentation to disease prediction; (iii) provide a starting point for people interested in experimenting and perhaps contributing to the field of machine learning for medical imaging by pointing out good educational resources, state-of-the-art open-source code, and interesting sources of data and problems related medical imaging. Artificial neural networksArtificial neural networks (ANNs) is one of the most famous machine learning models, introduced already in the 1950s, and actively studied since [41, Chapter 1.2]. 11 Roughly, a neural network consists of a number of connected computational units, called neurons, arranged in layers. There's an input layer where data enters the network, followed by one or more hidden layers transforming the data as it flows through, before ending at an output layer that produces the neural network's predictions. The network is trained to output useful predictions by identifying patterns in a set of labeled training data, fed through the network while the outputs are compared with the actual labels by an objective function. During training the network's parametersthe strength of each neuron-is tuned until the patterns identified by the network result in good 10 https://www.deeplearningbook.org/ 11 The loose connection between artificial neural networks and neural networks in the brain is often mentioned, but quite over-blown considering the complexity of biological neural networks. However, there is some interesting recent work connecting neuroscience and artificial neural networks, indicating an increase in the cross-fertilization between the two fields [43,44,45].3 predictions for the training data. Once the patterns are learned, the network can be used to make predictions on new, unseen data, i.e. generalize to new data.It h...

show abstract

Section: Image Synthesismentioning

confidence: 99%

An overview of deep learning in medical imaging focusing on MRI

Lundervold

2019

Zeitschrift für Medizinische Physik

1,518

883

View full text Add to dashboard Cite

show abstract

“…Comparatively, manipulation is an advanced form of generation, where besides understanding text, learning image semantics is compulsory to know the exact location of modification. Currently, image manipulation from GAN models is studied under different variations, from global [226] to local [213,215,221], directly from text [208,210,211,214,216] or with additional supervision [218,219,222,225], and from the latent space of GAN models [223,229].…”

Section: Supervised T2imentioning

confidence: 99%

“…The first study to purely explore image manipulation from GAN was by Dong et al [208]. They used a conditional GAN, following [7], where the generator encodes the input image to features and concatenates it with text semantics to decode the combined representation.…”

Section: Supervised T2imentioning

confidence: 99%

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

Ullah

Lee

et al. 2022

Sensors

View full text Add to dashboard Cite

For decades, co-relating different data domains to attain the maximum potential of machines has driven research, especially in neural networks. Similarly, text and visual data (images and videos) are two distinct data domains with extensive research in the past. Recently, using natural language to process 2D or 3D images and videos with the immense power of neural nets has witnessed a promising future. Despite the diverse range of remarkable work in this field, notably in the past few years, rapid improvements have also solved future challenges for researchers. Moreover, the connection between these two domains is mainly subjected to GAN, thus limiting the horizons of this field. This review analyzes Text-to-Image (T2I) synthesis as a broader picture, Text-guided Visual-output (T2Vo), with the primary goal being to highlight the gaps by proposing a more comprehensive taxonomy. We broadly categorize text-guided visual output into three main divisions and meaningful subdivisions by critically examining an extensive body of literature from top-tier computer vision venues and closely related fields, such as machine learning and human–computer interaction, aiming at state-of-the-art models with a comparative analysis. This study successively follows previous surveys on T2I, adding value by analogously evaluating the diverse range of existing methods, including different generative models, several types of visual output, critical examination of various approaches, and highlighting the shortcomings, suggesting the future direction of research.

show abstract

“…Meanwhile, there are also many works that have added condition information to the generative network and the discriminative network for conditional image synthesis. The condition information could be a discrete label [26,9,4,21,28], a reference image [24,10,40,8,6], or even a text sentence [31,41,10].…”

Section: Related Workmentioning

confidence: 99%

Towards Open-Set Identity Preserving Face Synthesis

Bao

Chen

Wen

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

248

172

View full text Add to dashboard Cite

We propose a framework based on Generative Adversarial Networks to disentangle the identity and attributes of faces, such that we can conveniently recombine different identities and attributes for identity preserving face synthesis in open domains. Previous identity preserving face synthesis processes are largely confined to synthesizing faces with known identities that are already in the training dataset. To synthesize a face with identity outside the training dataset, our framework requires one input image of that subject to produce an identity vector, and any other input face image to extract an attribute vector capturing, e.g., pose, emotion, illumination, and even the background. We then recombine the identity vector and the attribute vector to synthesize a new face of the subject with the extracted attribute. Our proposed framework does not need to annotate the attributes of faces in any way. It is trained with an asymmetric loss function to better preserve the identity and stabilize the training process. It can also effectively leverage large amounts of unlabeled training face images to further improve the fidelity of the synthesized faces for subjects that are not presented in the labeled training face dataset. Our experiments demonstrate the efficacy of the proposed framework. We also present its usage in a much broader set of applications including face frontalization, face attribute morphing, and face adversarial example detection.

show abstract

Semantic Image Synthesis via Adversarial Learning

Cited by 12 publications

References 0 publications

An overview of deep learning in medical imaging focusing on MRI

An overview of deep learning in medical imaging focusing on MRI

A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint

Towards Open-Set Identity Preserving Face Synthesis

Contact Info

Product

Resources

About