M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers

Abstract

Conditional image synthesis aims to create an image according to some multi-modal guidance in the forms of textual descriptions, reference images, and image blocks to preserve, as well as their combinations. In this paper, instead of investigating these control signals separately, we propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls. In UFC-BERT, both the diverse control signals and the synthesized image are uniformly represented as a sequence of discrete tokens to be processed by Transformer. Different from existing two-stage autoregressive approaches such as DALL-E and VQGAN, UFC-BERT adopts non-autoregressive generation (NAR) at the second stage to enhance the holistic consistency of the synthesized image, to support preserving specified image blocks, and to improve the synthesis speed. Further, we design a progressive algorithm that iteratively improves the non-autoregressively generated image, with the help of two estimators developed for evaluating the compliance with the controls and evaluating the fidelity of the synthesized image, respectively. Extensive experiments on a newly collected large-scale clothing dataset M2C-Fashion and a facial dataset Multi-Modal CelebA-HQ verify that UFC-BERT can synthesize high-fidelity images that comply with flexible multi-modal controls.Recently, two-stage image synthesis [42,48,3,13,47] has made great progress. The first stage learns a convolutional autoencoder with quantized latent representations for converting an image into a Preprint. Under review.

Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning

“…Our framework for multimodal video generation is a two-stage image generation method. It uses discrete feature representations [19,36,46,76]. During the first stage we train an autoencoder (with encoder E and decoder D) that has the same architecture as the one from VQGAN [19] to obtain a quantized representation for images.…”

“…Unlike existing transformerbased video generation works that focus on autoregressive training, we apply a non-autoregressive generation pipeline with a bidirectional transformer [20,21,23,33,65]. Our work is inspired by M6-UFC [76], which utilizes the nonautoregressive training for multimodal image generation and produces more diverse image generation with higher quality. Building upon M6-UFC, we further introduce training techniques for multimodal video synthesis.…”

“…Inspired by M6-UFC [76] and VIMPAC [55], we consider five masking strategies: (I) i.i.d. masking, i.e., randomly masking video tokens according to a Bernoulli distribution;…”

“…The first phase obtains discrete representations from images. We employ an autoencoder with a quantized bottleneck, inspired by the recent success of two-stage image generation using quantized feature representations [19,36,46,76]. The second phase learns to generate video representations that are conditioned on the input modalities, which can then be decoded into videos using the decoder from the first stage.…”

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

“…A possible solution is to discretize text, image, and object into a unified output vocabulary. Recent advances in image quantization [51,52] has demonstrated effectiveness in text-to-image synthesis [18,19,48,49], and thus we utilize this strategy for the target-side image representations. Sparse coding is effective in reducing the sequence length of image representation.…”