Cost-Effective Training in Low-Resource Neural Machine Translation

Abstract

While Active Learning (AL) techniques are explored in Neural Machine Translation (NMT), only a few works focus on tackling low annotation budgets where a limited number of sentences can get translated. Such situations are especially challenging and can occur for endangered languages with few human annotators or having cost constraints to label large amounts of data. Although AL is shown to be helpful with large budgets, it is not enough to build high-quality translation systems in these low-resource conditions. In this work, we propose a cost-effective training procedure to increase the performance of NMT models utilizing a small number of annotated sentences and dictionary entries. Our method leverages monolingual data with self-supervised objectives and a small-scale, inexpensive dictionary for additional supervision to initialize the NMT model before applying AL. We show that improving the model using a combination of these knowledge sources is essential to exploit AL strategies and increase gains in lowresource conditions. We also present a novel AL strategy inspired by domain adaptation for NMT and show that it is effective for low budgets. We propose a new hybrid data-driven approach, which samples sentences that are diverse from the labelled data and also most similar to unlabelled data. Finally, we show that initializing the NMT model and further using our AL strategy can achieve gains of up to 13 BLEU compared to conventional AL methods.

Train Global, Tailor Local: Minimalist Multilingual Translation into Endangered Languages

“…Random sampling is often surprisingly powerful (Kendall and Smith, 1938;Knuth, 1991;Sennrich et al, 2016a). There is extensive research to beat random sampling by methods based on entropy (Koneru et al, 2022), coverage and uncertainty (Peris and Casacuberta, 2018;Zhao et al, 2020), clustering Gangadharaiah et al, 2009), consensus , syntactic parsing (Miura et al, 2016), density and diversity (Koneru et al, 2022;Ambati et al, 2011), and learning to learn active learning strategies (Liu et al, 2018).…”

“…Let I l (•) and I r (•) be indicator functions to show whether a sentence belongs to the left or the right. We aim to maximize the diversity H c and optimize density by adjusting H l and H r (Koneru et al, 2022).…”

Geographical Distance Is The New Hyperparameter: A Case Study Of Finding The Optimal Pre-trained Language For English-isiZulu Machine Translation

“…Another work that utilizes backtranslation for effecctive NMT training is done by Dou et al (2020). Koneru et al (2022) proposes a cost-effective training procedure to increase the performance of models on NMT tasks, utilizing a small number of annotated sentences and dictionary entries. Park et al (2020) looked into decoding strategies for low-resourced languages in an attempt to improve training.…”