Emergent Communication Pretraining for Few-Shot Machine Translation

Abstract

While state-of-the-art models that rely upon massively multilingual pretrained encoders achieve sample efficiency in downstream applications, they still require abundant amounts of unlabelled text. Nevertheless, most of the world's languages lack such resources. Hence, we investigate a more radical form of unsupervised knowledge transfer in the absence of linguistic data. In particular, for the first time we pretrain neural networks via emergent communication from referential games. Our key assumption is that grounding communication on images-as a crude approximation of real-world environments-inductively biases the model towards learning natural languages. On the one hand, we show that this substantially benefits machine translation in few-shot settings. On the other hand, this also provides an extrinsic evaluation protocol to probe the properties of emergent languages ex vitro. Intuitively, the closer they are to natural languages, the higher the gains from pretraining on them should be. For instance, in this work we measure the influence of communication success and maximum sequence length on downstream performances. Finally, we introduce a customised adapter layer and annealing strategies for the regulariser of maximum-a-posteriori inference during fine-tuning. These turn out to be crucial to facilitate knowledge transfer and prevent catastrophic forgetting. Compared to a recurrent baseline, our method yields gains of 59.0%∼147.6% in BLEU score with only 500 NMT training instances and 65.1%∼196.7% with 1, 000 NMT training instances across four language pairs. These proofof-concept results reveal the potential of emergent communication pretraining for both natural language processing tasks in resource-poor settings and extrinsic evaluation of artificial languages.

Linking Emergent and Natural Languages via Corpus Transfer

“…We also establish the non-triviality of such a transfer performance by comparing to other synthetic and natural source corpora, as well as multiple ablation studies on the EC and downstream transfer setups to help understand the transferability of emergent language. Notably, our method of corpus transfer significantly outperforms directly transferring the trained emergent speaker model (Li et al, 2020b), demonstrating that modeling the emergent language could yield greater usefulness than directly transferring the EC agents.…”

“…A typical setup is the image referential game (Figure 1(a)), where a speaker generates a discrete sequence of tokens based on an input image, a listener is challenged to select the input out of distractors based on the message, and both networks are optimized jointly via game success signals. By studying these games, researchers are interested in the emergence of desirable properties resembling natural language, such as game success generalization Lazaridou & Baroni, 2020) and compositionality (Smith et al, 2003;Kirby et al, 2015;Lazaridou et al, 2018;Li et al, 2020b). However, these properties are mostly defined and analyzed within each individual game framework.…”

“…by pre-training a model on a corpus of emergent messages produced by a trained emergent speaker, and fine-tuning the model on downstream tasks with natural language data. This approach integrates inspirations from prior work like Papadimitriou & Jurafsky (2020), which proposes the transfer scheme from synthetic to natural corpora, and Li et al (2020b), which aims to improve few-shot translation by transferring the trained EC speaker and listener model weights. Through a series of experiments, we find that corpus transfer is helpful when the downstream natural language resource is limited.…”

“…There have been efforts to incorporate communication signals and natural language supervision to better communicate (Lowe et al, 2020), avoid language drift , or to improve NLP tasks like vision-language navigation (Fried et al, 2018), translation , and image captioning (Havrylov & Titov, 2017). Different from these setups where the speaker is also grounded on natural language annotations, Li et al (2020b) propose to separate the EC game, where no natural language is involved, and fine-tuning on a downstream translation task with limited natural language data. Despite being the closest work to ours, such a model transfer approach is fundamentally different from our corpus transfer in several ways.…”

“…Despite being the closest work to ours, such a model transfer approach is fundamentally different from our corpus transfer in several ways. First, while Li et al (2020b) mainly aim to improve downstream tasks, we envision transfer as a novel means to evaluating and analyzing emergent languages. Also, the emergent language is a representation with properties beyond the parameters of specific EC models (Section 3.1).…”