The Challenges of Large‐Scale, Web‐Based Language Datasets: Word Length and Predictability Revisited

Abstract

Language research has come to rely heavily on large-scale, web-based datasets. These datasets can present significant methodological challenges, requiring researchers to make a number of decisions about how they are collected, represented, and analyzed. These decisions often concern long-standing challenges in corpus-based language research, including determining what counts as a word, deciding which words should be analyzed, and matching sets of words across languages. We illustrate these challenges by revisiting "Word lengths are optimized for efficient communication" (Piantadosi, Tily, & Gibson, 2011), which found that word lengths in 11 languages are more strongly correlated with their average predictability (or average information content) than their frequency. Using what we argue to be best practices for large-scale corpus analyses, we find significantly attenuated support for this result and demonstrate that a stronger relationship obtains between word frequency and length for a majority of the languages in the sample. We consider the implications of the results for language research more broadly and provide several recommendations to researchers regarding best practices.

Frequency, informativity and word length: Insights from typologically diverse corpora

“…Each corpus contained 10 million sentences collected from diverse online sources and randomly shuffled by the corpus creators. The UTF-8 encoding was used, in accordance with [5], to make sure that strings like Spanish si 'if' and sí 'yes' are treated as different words.…”

“…Uppercase characters were converted to lowercase. After that, the unigrams were cleaned: following the filtering methods in [5], I retained only those strings that occurred in the corresponding OpenSubtitles corpus 1 [25] and in the Hunspell dictionary (with the exception of Finnish, for which the Microsoft spellchecker was used). The removed strings represented words written in a different language (most importantly, English), misspellings, proper names, acronyms and punctuation marks.…”

“…Informativity was based on conditional probabilities of words given one, two or three preceding words. At the same time, a more recent study in [5] has demonstrated that the dominance of contextual informativity is no longer observed if other methodological choices are made: in particular, if one encodes strings in UTF-8, rather than in the ASCII standard, and excludes words that do not occur in spelling dictionaries and in the OPUS corpus of film subtitles [23]. More exactly, only two Google 1T n-gram datasets used in [4], representing English and French, have shown higher correlations of informativity and word length, after the dictionary and subtitles filters have been applied to the target words.…”

“…Finally, a methodological goal of this study is to extend the cleaning method in [5] to context n-grams and check if the bigrams we select for analysis lead to different correlations of the resulting informativity measure with word length.…”

“…

Zipf's law of abbreviation, which posits a negative correlation between word frequency and length, is one of the most famous and robust cross-linguistic generalizations [1][2][3]. At the same time, it has been shown that contextual informativity (average surprisal given previous context) can be more strongly correlated with word length [4], although this tendency is not observed consistently, depending on several methodological choices [5]. The present study, which examines a more diverse sample of languages than in the previous studies (Arabic, Finnish, Hungarian, Indonesian, Russian, Spanish and Turkish), reveals intriguing crosslinguistic differences, which can be explained by typological properties of the languages.

…”