On the Importance of Subword Information for Morphological Tasks in Truly Low-Resource Languages

Abstract

Recent work has validated the importance of subword information for word representation learning. Since subwords increase parameter sharing ability in neural models, their value should be even more pronounced in low-data regimes. In this work, we therefore provide a comprehensive analysis focused on the usefulness of subwords for word representation learning in truly low-resource scenarios and for three representative morphological tasks: fine-grained entity typing, morphological tagging, and named entity recognition. We conduct a systematic study that spans several dimensions of comparison: 1) type of data scarcity which can stem from the lack of taskspecific training data, or even from the lack of unannotated data required to train word embeddings, or both; 2) language type by working with a sample of 16 typologically diverse languages including some truly low-resource ones (e.g. Rusyn, Buryat, and Zulu); 3) the choice of the subword-informed word representation method. Our main results show that subword-informed models are universally useful across all language types, with large gains over subword-agnostic embeddings. They also suggest that the effective use of subwords largely depends on the language (type) and the task at hand, as well as on the amount of available data for training the embeddings and taskbased models, where having sufficient in-task data is a more critical requirement.

Transliteration for Cross-Lingual Morphological Inflection

“…The majority of the world's languages are synthetic, meaning they have rich morphology. As a result, modeling morphological inflection computationally can have a significant impact on downstream quality, not only in analysis tasks such as named entity recognition and morphological analysis (Zhu et al, 2019), but also for language generation systems for morphologically-rich languages.…”

Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

“…to construct the final word vector, and conclude that the best performing configuration is highly language and task dependent. A subsequent work (Zhu et al, 2019a) focuses on LRLs and finds the combination of BPE and addition largely robust, although they once again note language-dependent variability. They also find that encoding "affix" information with positional embeddings is beneficial, hinting that the embedding space may distinguish the importance of different kinds of subwords.…”

An Assessment of the Impact of OCR Noise on Language Models

“…BERT, for example, has been proven sensitive to (non-adversarial) human noise Sun et al (2020); Kumar et al (2020). Examples of models that can be more resilient to noise include typological language models Gerz et al (2018); Ponti et al (2019), sub-word or character-level language models Kim et al (2016); Zhu et al (2019); Ma et al (2020), byte-pair encoding Sennrich et al (2016), and their extension in recent tokenization-free models (Heinzerling and Strube, 2018;Clark et al, 2021;Xue et al, 2021), yet their use as noise-resilient language models remains to be fully assessed.…”