A Simple Recipe for Multilingual Grammatical Error Correction

Abstract

This paper presents a simple recipe to train state-of-the-art multilingual Grammatical Error Correction (GEC) models. We achieve this by first proposing a language-agnostic method to generate a large number of synthetic examples. The second ingredient is to use largescale multilingual language models (up to 11B parameters). Once fine-tuned on languagespecific supervised sets we surpass the previous state-of-the-art results on GEC benchmarks in four languages: English, Czech, German and Russian. Having established a new set of baselines for GEC, we make our results easily reproducible and accessible by releasing a CLANG-8 dataset. 1 It is produced by using our best model, which we call gT5, to clean the targets of a widely used yet noisy LANG-8 dataset. CLANG-8 greatly simplifies typical GEC training pipelines composed of multiple fine-tuning stages -we demonstrate that performing a single fine-tuning step on CLANG-8 with the off-the-shelf language models yields further accuracy improvements over an already top-performing gT5 model for English.

Correcting diacritics and typos with a ByT5 transformer model

“…A limitation of our work is that we had only a single moderate GPU at our disposal. Scaling model size [105], incorporating additional datasets [46], and training longer can improve accuracy by several percent. Similarly, one can build a model of multiple languages to gain benefits by overlapping vocabularies and semantics of related under-represented languages, although studies report contradictory results [48,46].…”

“…The popular seq2seq transformer T5 [99] used batch size 128 for both pre-training and fine-tuning. Follow-up models such as the multilingual version mT5 [100], the grammatical error correction model gT5 [105], and ByT5 [8] (the one we use in this work) all carried on with the same value for fine-tuning. The same size is also used in works solving the diacritics restoration task [47,106].…”

“…However, due to the higher "energy" (or "temperature") in the optimization the high η causes "bouncing" of parameter values and prevents settling in the best spot, resulting in the higher final training loss. An optimal learning rate value, as used during fine-tuning of the T5 family models [99,105,100,8] with Adafactor optimizer, is 0.001.…”

ÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5

“…Grundkiewicz et al (2019) approach GEC as a neural machine translation task using the Transformer architecture (Vaswani et al, 2017), which is pre-trained using a vast amount of synthetic data generated by character-level and word-level edits. Recently, Rothe et al (2021) presented a GEC system based on multilingual mT5 (Xue et al, 2021b), reaching state-of-the-art results on several datasets with the gigantic xxl model size with 13B parameters.…”

“…For lexical normalization, we could directly use the unnormalized sentence as input and the normalized sentence as output (an approach used by Rothe et al (2021) for GEC). However, we were concerned that such an approach would be too different from the ByT5 pre-training, and furthermore, it would not allow to reconstruct the alignment of the normalized tokens when a word is removed during normalization or split into several words.…”