Controlling Style and Semantics in Weakly-Supervised Image Generation

Pavllo, Dario; Lucchi, Aurélien; Hofmann, Thomas

doi:10.1007/978-3-030-58539-6_29

Cited by 34 publications

(21 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, our method can generate complex scene images containing multiple objects from a reconfigurable layout. Most similar to our work are the methods proposed by Ke Ma et al [27] as an extension of [27] using an auxiliary attribute classifier and explicit reconstruction loss for horizontally shifted objects, and [32] which requires semantic instance masks. To the best of our knowledge, [27] is currently the only other direct layout-to-image method using attributes.…”

Section: Usage Of Attributesmentioning

confidence: 76%

AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

Фролов¹,

Sharma²,

Hees³

et al. 2021

Preprint

View full text Add to dashboard Cite

Conditional image synthesis from layout has recently attracted much interest. Previous approaches condition the generator on object locations as well as class labels but lack fine-grained control over the diverse appearance aspects of individual objects. Gaining control over the image generation process is fundamental to build practical applications with a user-friendly interface. In this paper, we propose a method for attribute controlled image synthesis from layout which allows to specify the appearance of individual objects without affecting the rest of the image. We extend a state-of-the-art approach for layout-to-image generation to additionally condition individual objects on attributes. We create and experiment on a synthetic, as well as the challenging Visual Genome dataset. Our qualitative and quantitative results show that our method can successfully control the fine-grained details of individual objects when modelling complex scenes with multiple objects.

show abstract

Section: Usage Of Attributesmentioning

confidence: 76%

AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

Фролов¹,

Sharma²,

Hees³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Since the introduction of using BiLSTM in AttnGAN [35] to encode captions, most of the following works adopted it. However, recent works [37,38] leverage pre-trained transformer-based models such as BERT [39] to obtain text embeddings.…”

Section: Encoding Textmentioning

confidence: 99%

“…Pavllo et al [38] proposed a weakly-supervised approach by exploiting sparse, instance semantic masks. In contrast to dense pixel-based masks, sparse instance masks allow easy editing operations such as adding or removing objects because the user does not face the problem of "filling in wholes".…”

Section: Semantic Masksmentioning

confidence: 99%

Adversarial Text-to-Image Synthesis: A Review

Frolov,

Hinz,

Raue

et al. 2021

Preprint

View full text Add to dashboard Cite

With the advent of generative adversarial networks, synthesizing images from textual descriptions has recently become an active research area. It is a flexible and intuitive way for conditional image generation with significant progress in the last years regarding visual realism, diversity, and semantic alignment. However, the field still faces several challenges that require further research efforts such as enabling the generation of high-resolution images with multiple objects, and developing suitable and reliable evaluation metrics that correlate with human judgement. In this review, we contextualize the state of the art of adversarial text-to-image synthesis models, their development since their inception five years ago, and propose a taxonomy based on the level of supervision. We critically examine current strategies to evaluate textto-image synthesis models, highlight shortcomings, and identify new areas of research, ranging from the development of better datasets and evaluation metrics to possible improvements in architectural design and model training. This review complements previous surveys on generative adversarial networks with a focus on text-to-image synthesis which we believe will help researchers to further advance the field.

show abstract

“…Image methods. Controllable generative models have also been developed for images (Härkönen et al, 2020;Esser et al, 2019;Singh et al, 2019;Lample et al, 2017;Karras et al, 2020;Brock et al, 2019;Collins et al, 2020;Shen et al, 2020;Esser et al, 2020;Goetschalckx et al, 2019;Pavllo et al, 2020;, which control the object class, pose, lighting, etc., of an image. Many image style transform methods have also been developed Gatys et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Chang¹,

Shrivastava²,

Koppula³

et al. 2021

Preprint

View full text Add to dashboard Cite

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the traininginference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles. 1 * work done during internship at Apple 1 Our synthesis results can be viewed at https://apple.github.io/ml-style-equalization

show abstract

Controlling Style and Semantics in Weakly-Supervised Image Generation

Cited by 34 publications

References 28 publications

AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

AttrLostGAN: Attribute Controlled Image Synthesis from Reconfigurable Layout and Style

Adversarial Text-to-Image Synthesis: A Review

Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Contact Info

Product

Resources

About