ZstGAN: An adversarial approach for Unsupervised Zero-Shot Image-to-image Translation

Lin, Jianxin; Xia, Yingce; Liu, Sen; Zhao, Shuxin; Chen, Zhibo

doi:10.1016/j.neucom.2021.07.037

Cited by 17 publications

(3 citation statements)

References 31 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al. [37] used a GANbased model to learn a multi-modal consistent semantic representation, and the disentangled domain-invariant features are extracted for unsupervised zero-shot image-to-image translation.…”

Section: Related Workmentioning

confidence: 99%

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Wang¹,

Yingyu²,

Sermesant³

et al. 2023

Preprint

View full text Add to dashboard Cite

Cross-modality data translation has attracted great interest in image computing. Deep generative models (e.g., GANs) show performance improvement in tackling those problems. Nevertheless, as a fundamental challenge in image translation, the problem of Zero-shot-Learning Cross-Modality Data Translation with fidelity remains unanswered. This paper proposes a new unsupervised zero-shot-learning method named Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion), which learns to translate the unseen source data to the target domain. The MIDiffusion leverages a score-matching-based generative model, which learns the prior knowledge in the target domain. We propose a differentiable local-wise-MI-Layer (LM I) for conditioning the iterative denoising sampling. The LM I captures the identical cross-modality features in the statistical domain for the diffusion guidance; thus, our method does not require retraining when the source domain is changed, as it does not rely on any direct mapping between the source and target domains. This advantage is critical for applying cross-modality data translation methods in practice, as a reasonable amount of source domain dataset is not always available for supervised training. We empirically show the advanced performance of MIDiffusion in comparison with an influential group of generative models, including adversarial-based and other scorematching-based models.

show abstract

Section: Related Workmentioning

confidence: 99%

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Wang¹,

Yingyu²,

Sermesant³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…ZstGAN [28], the closest neighbour, requires many source domains (that could for example be extracted from image labels of one dataset). The highest resolution handled in that work is 128 2 , which is signifantly lower than our method.…”

Section: Zero-shot Domain Transfermentioning

confidence: 99%

High Resolution Zero-Shot Domain Adaptation of Synthetically Rendered Face Images

Garbin¹,

Kowalski²,

Johnson³

et al. 2020

Preprint

View full text Add to dashboard Cite

Generating photorealistic images of human faces at scale remains a prohibitively difficult task using computer graphics approaches. This is because these require the simulation of light to be photorealistic, which in turn requires physically accurate modelling of geometry, materials, and light sources, for both the head and the surrounding scene. Non-photorealistic renders however are increasingly easy to produce. In contrast to computer graphics approaches, generative models learned from more readily available 2D image data have been shown to produce samples of human faces that are hard to distinguish from real data. The process of learning usually corresponds to a loss of control over the shape and appearance of the generated images. For instance, even simple disentangling tasks such as modifying the hair independently of the face, which is trivial to accomplish in a computer graphics approach, remains an open research question. In this work, we propose an algorithm that matches a non-photorealistic, synthetically generated image to a latent vector of a pretrained StyleGAN2 model which, in turn, maps the vector to a photorealistic image of a person of the same pose, expression, hair, and lighting. In contrast to most previous work, we require no synthetic training data. To the best of our knowledge, this is the first algorithm of its kind to work at a resolution of 1K and represents a significant leap forward in visual realism.

show abstract

“…Image-to-image translation (I2IT) [1] is proposed to visually transform images of one style into another and has attracted a great deal of attention due to its extensive application in the fields of style transfer [2], image colorization [3], remote sensing [4][5][6], target detection [7], data representation [8,9], underwater image restoration [10], medical image processing [11,12], haze removal [13] and noise removal [14], etc. Following several years of development, researchers have found that generative adversarial networks [15] and their variant models are effective solutions for most image translation tasks and obtain very impressive results in both supervised and unsupervised [16,17] settings.…”

Section: Introductionmentioning

confidence: 99%

Binary Noise Guidance Learning for Remote Sensing Image-to-Image Translation

Zhang,

Zhou,

Zheng

et al. 2023

Remote Sensing

View full text Add to dashboard Cite

Image-to-image translation (I2IT) is an important visual task that aims to learn a mapping of images from one domain to another while preserving the representation of the content. The phenomenon known as mode collapse makes this task challenging. Most existing methods usually learn the relationship between the data and latent distributions to train more robust latent models. However, these methods often ignore the structural information among latent variables, leading to patterns in the data being obscured during the process. In addition, the inflexibility of data modes caused by ignoring the latent mapping of two domains is also one of the factors affecting the performance of existing methods. To make the data schema stable, this paper develops a novel binary noise guidance learning (BnGLGAN) framework for image translation to solve these problems. Specifically, to eliminate uncertainty of domain distribution, a noise prior inference learning (NPIL) module is designed to infer an estimated distribution from a certain domain. In addition, to improve the authenticity of reconstructed images, a distribution-guided noise reconstruction learning (DgNRL) module is introduced to reconstruct the noise from the source domain, which can provide source semantic information to guide the GAN’s generation. Extensive experiments fully prove the efficiency of our proposed framework and its advantages over comparable methods.

show abstract

ZstGAN: An adversarial approach for Unsupervised Zero-Shot Image-to-image Translation

Cited by 17 publications

References 31 publications

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

High Resolution Zero-Shot Domain Adaptation of Synthetically Rendered Face Images

Binary Noise Guidance Learning for Remote Sensing Image-to-Image Translation

Contact Info

Product

Resources

About