Multi-task learning for historical text normalization: Size matters

Abstract

Historical text normalization suffers from small datasets that exhibit high variance, and previous work has shown that multitask learning can be used to leverage data from related problems in order to obtain more robust models. Previous work has been limited to datasets from a specific language and a specific historical period, and it is not clear whether results generalize. It therefore remains an open problem, when historical text normalization benefits from multi-task learning. We explore the benefits of multi-task learning across 10 different datasets, representing different languages and periods. Our main findingcontrary to what has been observed for other NLP tasks-is that multi-task learning mainly works when target task data is very scarce.

Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set

“…Data. We experiment on the ten datasets from Bollmann et al (2018), which represent eight different languages: German (two datasets; Bollmann et al, 2017;Odebrecht et al, 2017); English, Hungarian, Icelandic, and Swedish (Pettersson, 2016); Slovene (two datasets; Ljubešic et al, 2016); and Spanish and Portuguese (Vaamonde, 2015). We treat the two datasets for German and Slovene as different languages.…”

“…Both encoder and decoder have a single hidden layer. We use the default model in Open-NMT (Klein et al, 2017) 3 as our implementation and employ the hyperparameters from Bollmann et al (2018). In the original paper, early stopping is done by training for 50 epochs, and the best model regarding development accuracy is applied to the test set.…”

“…However, since T is finite, overfitting the training set might lead to poor generalization performance. One way to avoid fitting Equation 1 too # train # dev ES Bollmann et al (2018) 5k 12k-46k Yes 400-700 100-200 Yes Makarov and Clematide (2018) 100 1k Yes Sharma et al (2018) 100 100 Yes Schulz et al (2018) 1k-21k 9k N/A Upadhyay et al (2018) 500 1k Yes closely is early stopping: a separate development or validation set is used to end training as soon as the loss on the development set L D (θ) starts increasing or model performance on the development set D starts decreasing. The best set of parameters θ is used in the final model.…”

“…This leads to settings where validation examples may outnumber training examples. Table 1 shows such cases for the tasks of historical text normalization (Bollmann et al, 2018), morphological segmentation , morphological inflection (Makarov and Clematide, 2018;Sharma et al, 2018), argument component identification (Schulz et al, 2018), and transliteration (Upadhyay et al, 2018).…”

Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD

“…One active research question however is what information in specific tasks should be shared, as well was what indicators can be used to predetermine the cost-benefit trade-offs of MTL for a given application. Findings have shown that label distributions (Martínez Alonso and Plank, 2017), data sizes (Bollmann et al, 2018) and single task loss curves (Bingel and Søgaard, 2017) have all been respective indicators for MTL performance. Different tasks, data sizes, and settings can all show different relative performance gains (Adouane and Bernardy, 2020).…”