On Language Models for Creoles

Abstract

Creole languages such as Nigerian Pidgin English and Haitian Creole are under-resourced and largely ignored in the NLP literature. Creoles typically result from the fusion of a foreign language with multiple local languages, and what grammatical and lexical features are transferred to the creole is a complex process (Sessarego, 2020). While creoles are generally stable, the prominence of some features may be much stronger with certain demographics or in some linguistic situations (Winford, 1999;Patrick, 1999). This paper makes several contributions: We collect existing corpora and release models for Haitian Creole, Nigerian Pidgin English, and Singaporean Colloquial English. We evaluate these models on intrinsic and extrinsic tasks. Motivated by the above literature, we compare standard language models with distributionally robust ones and find that, somewhat surprisingly, the standard language models are superior to the distributionally robust ones. We investigate whether this is an effect of overparameterization or relative distributional stability, and find that the difference persists in the absence of over-parameterization, and that drift is limited, confirming the relative stability of creole languages.

Evaluating Distributional Distortion in Neural Language Modeling

“…We additionally contribute to the body work seeking to characterize and adapt neural model performance on rare or novel examples and classes (Horn & Perona, 2017;Bengio, 2015). In the context of language modeling, Lent et al (2021) explored performance on under-resourced languages, whereas Oren et al (2019) did so on under-represented domains in training corpora. Mc-Coy et al (2021) introduced analyses to assess sequential and syntactic novelty in LMs.…”

“…Meister & Cotterell (2021), for example, investigated the statistical tendencies of the distribution defined by neural LMs, whereas Kulikov et al (2021) explored whether they adequately capture the modes of the distribution they attempt to model. At the same time, increased focus has been given to performance on rare or novel events in the data distribution, both for models of natural language (McCoy et al, 2021;Lent et al, 2021;Dudy & Bedrick, 2020;Oren et al, 2019) and neural models more generally (see, for example Sagawa et al, 2020;D'souza et al, 2021;Blevins & Zettlemoyer, 2020;Czarnowska et al, 2019;Horn & Perona, 2017;Ouyang et al, 2016;Bengio, 2015;Zhu et al, 2014). Neither of these branches of work, however, has explored instancelevel LM performance on rare sequences in the distribution.…”

What a Creole Wants, What a Creole Needs

“…Hagemeijer et al (2014) presents an extensive overview of Creole data resources through 2014 for a wide variety of Creoles, many of which are more traditional corpora, (e.g., transcriptions of conversations made by linguists with formal training, or scans of documents originally written in the Creole language); though these may not have the relevant annotations for common NLP tasks. Lent et al (2021) also provides a thorough overview of existing NLP datasets for Haitian Kreyol, Singaporean Colloquial English (Singlish), and Nigerian Pidgin English. In this work, we set about the task of manually verifying each dataset presented by Hagemeijer et al (2014) and Lent et al (2021), as well as searching for additional resources.…”

“…Lent et al (2021) also provides a thorough overview of existing NLP datasets for Haitian Kreyol, Singaporean Colloquial English (Singlish), and Nigerian Pidgin English. In this work, we set about the task of manually verifying each dataset presented by Hagemeijer et al (2014) and Lent et al (2021), as well as searching for additional resources. We present all "verified" datasets in Table 1.…”

“…Notably, Murawaki (2016) use APiCS features (Michaelis et al, 2013) to encode Creoles, and utilize different approaches for language evolution modeling, to reach the final conclusion that Creoles are not typologically distinct from non-Creole languages. Meanwhile, Lent et al (2021) explored the question of how to effectively build language models for three Creole languages (Haitian Kreyol, Singaporean Colloquial English, and Nigerian Pidgin). Their approach involved experimenting with distributionally robust objectives (Oren et al, 2019), to ascertain whether data from a Creole's "parent" languages could help the language model to be more robust.…”