A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation

Abstract

Recent advances in the pre-training of language models leverage large-scale datasets to create multilingual models. However, low-resource languages are mostly left out in these datasets. This is primarily because many widely spoken languages are not well represented on the web and therefore excluded from the large-scale crawls used to create datasets. Furthermore, downstream users of these models are restricted to the selection of languages originally chosen for pre-training. This work investigates how to optimally leverage existing pre-trained models to create low-resource translation systems for 16 African languages. We focus on two questions: 1) How can pre-trained models be used for languages not included in the initial pre-training? and 2) How can the resulting translation models effectively transfer to new domains? To answer these questions, we create a new African news corpus covering 16 languages, of which eight languages are not part of any existing evaluation dataset. We demonstrate that the most effective strategy for transferring both to additional languages and to additional domains is to fine-tune large pre-trained models on small quantities of highquality translation data.

Better Quality Pre-training Data and T5 Models for African Languages

“…While Kreutzer et al (2022)'s audit of mC4 did not yield a significant amount of offensive content (0.06% of sentences they audited) and our web crawls mainly focused on verified news publications, these filters ensure that non-linguistic and offensive contents are removed at the passage level.…”

“…Previous works have shown the correlation between the quality of the data used in pretraining a model and the performance of the trained model (Rae et al, 2021;Kreutzer et al, 2022;Hernandez et al, 2022). AfriTeVa V2's improvement over baselines in downstream tasks suggests that this is true.…”

“…The introduction of mC4 (Xue et al, 2021), a document-level dataset spanning 101 languages helped alleviate this cover-age gap. 1 However, previous work (Kreutzer et al, 2022) has shown that mC4 and other existing largescale pretraining corpora have numerous quality issues, particularly for the low-resource African languages they contain.…”

“…Motivated by this need, we introduce a new multilingual pretraining corpus in 20 African languages. We draw from Kreutzer et al (2022)'s audit of existing pretraining corpora to understand prevailing quality issues. For mC4, they cite a high ratio both of sentences in incorrect languages (15.98% average) and nonlinguistic content (11.40% average).…”

Cross-lingual Sentiment Analysis via AAE and BiGRU

“…If the quality of the translator is bad, the model can even underperform the FT baseline without any cross-lingual transfer. Considering that most low-resource languages do not have high-quality MT systems yet (Adelani et al, 2022), this further implies we should rather focus on translation-free approaches for this task.…”