A large quantitative analysis of written language challenges the idea that all languages are equally complex

Abstract

One of the fundamental questions about human language is whether all languages are equally complex. Here, we approach this question from an information-theoretic perspective. We present a large scale quantitative cross-linguistic analysis of written language by training a language model on more than 6,500 different documents as represented in 41 multilingual text collections consisting of ~3.5 billion words or ~9.0 billion characters and covering 2,069 different languages that are spoken as a native language by more than 90% of the world population. We statistically infer the entropy of each language model as an index of what we call average prediction complexity. We compare complexity rankings across corpora and show that a language that tends to be more complex than another language in one corpus also tends to be more complex in another corpus. In addition, we show that speaker population size predicts entropy. We argue that both results constitute evidence against the equi-complexity hypothesis from an information-theoretic perspective.

Languages with more speakers tend to be harder to (machine-)learn

“…faster) to learn: in a large-scale quantitative cross-linguistic analysis, ref. 25 trained an LM on more than 6,500 documents in over 2,000 different languages and statistically inferred the entropy rate of each document, which can be seen as an index of the underlying language complexity. 28,[33][34][35] The results showed that documents in languages with more speakers tended to be more complex.…”

“…In this article, we first harness part of the data used by ref. 25 to explicitly test this hypothesis. Since the LM used by ref.…”

“…Since the LM used by ref. 25 is rather simple, we train two more sophisticated LMs that use machine-and deep-learning on the data and compare results. We then discuss some potential limitations of the multilingual text collection used by ref.…”

“…We then discuss some potential limitations of the multilingual text collection used by ref. 25 . To rule out that the results are driven by these limitations, we create two fully balanced and parallel multilingual corpora, which we use to train seven different LMsranging from very simple n-gram models to state-of-the-art deep neural networksand measure how difficult it is for each LM to build an adequate representation of the input.Importantly, previous research [36][37][38][39][40][41]21 has shown that cross-linguistic (and cross-cultural) studies that seek to analyse potential statistical associations between language features and external factors must take into account Galton's problem, which refers to the potential confounding of…”

“…We then discuss some potential limitations of the multilingual text collection used by ref. 25 . To rule out that the results are driven by these limitations, we create two fully balanced and parallel multilingual corpora, which we use to train seven different LMsranging from very simple n-gram models to state-of-the-art deep neural networksand measure how difficult it is for each LM to build an adequate representation of the input.…”