The Entropy of Words—Learnability and Expressivity across More Than 1000 Languages

Abstract

Abstract:The choice associated with words is a fundamental property of natural languages. It lies at the heart of quantitative linguistics, computational linguistics, and language sciences more generally. Information-theory gives us tools at hand to measure precisely the average amount of choice associated with words -the word entropy. Here we use three parallel corpora -encompassing ca. 450 million words in 1916 texts and 1259 languages -to tackle some of the major conceptual and practical problems of word entropy estimation: dependence on text size, register, style and estimation method, as well as non-independence of words in co-text. We present three main results: 1) a text size of 50K tokens is sufficient for word entropies to stabilize throughout the text, 2) across languages of the world, word entropies display a unimodal distribution that is skewed to the right. This suggests that there is a trade-off between the learnability and expressivity of words across languages of the world. 3) There is a strong linear relationship between unigram entropies and entropy rates, suggesting that they are inherently linked. We discuss the implications of these results for studying the diversity and evolution of languages from an information-theoretic point of view.

Does Zipf’s law of abbreviation shape birdsong?

“…If users must convey an idea only infrequently, then they can invest effort in longer words to ensure that the idea is communicated clearly [14]. Evidence supporting ZLA has been found in each of the nearly 1,000 human languages where it has been sought [15], and the law applies to both spoken language [16, 17] and written characters [18, 19]. Researchers have reported mixed support for ZLA in the vocal communication of other animals [20], including primates [17, 21–25], cetaceans [26], bats [27], and hyraxes [28].…”

“…First, relative to the number of words in human languages, the number of note types used by most bird populations is small. A small number of note types makes it more difficult to detect a significant concordance between note type frequency and duration [15, 17]. If the number of note types is very small, as in the calls of African penguins [34], then even a perfect concordance between the frequency and duration of note types may provide only weak evidence for or against ZLA.…”

“…Thus, it may be reasonable to study ZLA at the population level. Researchers can select representative multiauthor texts and assess the concordance between the frequency and duration for each word using simple rank correlations [15]. In contrast, in many bird species, individuals in the same population use different and sometimes non-overlapping sets of note types [40].…”

“…Finally, perhaps the biggest challenge to studying ZLA in birdsong arises from the inherent difficulty of classifying notes. In human languages, especially in the written form, we can usually agree on whether two units represent the same word or different words [13, 15, 45]. In birdsong, determining whether two notes belong to the same note type is less straightforward.…”

“…Nonetheless, we believe such data is worth analysing, and we encourage authors to do so when they have appropriate data available. If we find little evidence for ZLA in birdsong, or if we find many populations in which the duration and the frequency of use of notes are positively correlated, then we might conclude that ZLA does not operate in birdsong, or at least that it is not universal in birdsong, as it appears to be in human language [15]. If we find widespread evidence for ZLA, then researchers may consider more focused studies to identify the potential mechanisms that produce ZLA.…”