X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Abstract

Mirroring the success of masked language models, vision-and-language counterparts like VILBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual question answering and visual grounding. Recent work has also successfully adapted such models towards the generative task of image captioning. This begs the question: Can these models go the other way and generate images from pieces of text? Our analysis of a popular representative from this model family -LXMERT -finds that it is unable to generate rich and semantically meaningful imagery with its current training setup. We introduce X-LXMERT, an extension to LXMERT with training refinements including: discretizing visual representations, using uniform masking with a large range of masking ratios and aligning the right pre-training datasets to the right objectives which enables it to paint. X-LXMERT's image generation capabilities rival state of the art generative models while its question answering and captioning abilities remains comparable to LXMERT. Finally, we demonstrate the generality of these training refinements by adding image generation capabilities into UNITER to produce X-UNITER.

Die Enkel von Locard

“…Dazu gehörten beispielsweise die Objekterkennung und somit die Verbindung zwischen bildlichen und textlichen Darstellungen. Die semantische Analyse von Bildern oder Videos wird dabei bis heute beforscht [33,34]. Image GPT ist dabei ein aktuelles Beispiel, wie anhand von Bildern einem System Bezeichner, auch Labels genannt, angelernt werden können.…”

A Picture is Worth a Thousand Words

“…The Transformer accepts a sequence of image and text representations as inputs, encodes them to contextualized vector representations, and outputs image and text tokens. For text-toimage generation, we follow X-LXMERT [2] to use an GAN-based image generator to convert image tokens to a real scene image.…”

“…We use the original grid features as visual inputs for image-to-text generation tasks to reduce the loss of image information. We adopt discrete clustering features of the original features to construct the ground-truth visual tokens as output prediction for text-to-image generation [2].…”

“…During training, we sample a masking ratio from a uniform prior distribution ([0,1]) and randomly mask the percentage of target tokens for prediction. While in inference, we adopt a non-autoregressive sampling strategy (i.e., mask-predict-k strategy [2,3,5]). So only a few sampling steps (e.g., 4) are needed to generate all target tokens, which enable real-time inference.…”

Unifying Multimodal Transformer for Bi-directional Image and Text Generation

“…(1) Task-specific uni-directional architectures [2,6,53] on bi-directional image and text generation tasks. Our taskagnostic bi-directional architecture as show in (2) releases design efforts of task-specific architectures in (1).…”