CM3: A Causal Masked Multimodal Model of the Internet

Abstract

We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked languageimage models on large-scale web and Wikipedia articles, where each document contains all of the text, hypertext markup, hyperlinks, and image tokens (from a VQVAE-GAN), provided in the order they appear in the original HTML source (before masking). The resulting CM3 models can generate rich structured, multimodal outputs while conditioning on arbitrary masked document contexts, and thereby implicitly learn a wide range of text, image, and cross modal tasks. They can be prompted to recover, in a zero-shot fashion, the functionality of models such as DALL-E, GENRE, and HTLM De Cao et al., 2020;Aghajanyan et al., 2021). We set the new state-of-the-art in zero-shot summarization, entity linking, and entity disambiguation while maintaining competitive performance in the fine-tuning setting. We can generate images unconditionally, conditioned on text (like DALL-E) and do captioning all in a zero-shot setting with a single model.

InCoder: A Generative Model for Code Infilling and Synthesis

“…We compare the validation perplexity of the 6B parameter model and a smaller 1.3B parameter model (see Section 6.2 for details on the training of this 1.3B model) in Figure 2, showing comparable scaling laws to those reported by Aghajanyan et al [2] HumanEval Pass@1 We plot a line of best fit along with a 95% confidence interval via bootstrap resampling.…”

“…On the other hand, masked language models can condition on both the left and right contexts to infill a masked region, however, their training objective is typically limited to generating only about 15% of a document. In this paper, we adopt the recently proposed causal masking objective [2], which aims to combine the strengths of both causal and masked language models.…”

“…We perform one epoch on the training data, using each training document exactly once. Our implementation utilized the causal masking implementation [2] available in Fairseq [47], with the underlying library being PyTorch [49]. Our per-GPU batch size was 8, with a maximum token sequence length of 2048.…”

“…Like prior work, INCODER is trained to maximize the likelihood of a corpus of code. However, we adopt a causal masking objective [2], allowing INCODER to infill blocks of code conditioned on arbitrary left and right contexts.…”

Hierarchical Text-Conditional Image Generation with CLIP Latents

“…Many previous works have trained GANs [21] on publicly available image captioning datasets to produce text-conditional image samples [56,63,49,58,57]. Other works have adapted the VQ-VAE approach [52] to text-conditional image generation by training autoregressive transformers on sequences of text tokens followed by image tokens [40,12,1]. Finally, some works have applied diffusion models to the problem, training either continuous [35] or discrete [22] diffusion models with auxiliary text encoders to handle textual input.…”