Challenges in detecting evolutionary forces in language change using diachronic corpora

Abstract

Newberry et al. (Detecting evolutionary forces in language change, Nature 551, 2017) tackle an important but difficult problem in linguistics, the testing of selective theories of language change against a null model of drift. Having applied a test from population genetics (the Frequency Increment Test) to a number of relevant examples, they suggest stochasticity has a previously under-appreciated role in language evolution. We replicate their results and find that while the overall observation holds, results produced by this approach on individual time series can be sensitive to how the corpus is organized into temporal segments (binning). Furthermore, we use a large set of simulations in conjunction with binning to systematically explore the range of applicability of the Frequency Increment Test. We conclude that care should be exercised with interpreting results of tests like the Frequency Increment Test on individual series, given the researcher degrees of freedom available when applying the test to corpus data, and fundamental differences between genetic and linguistic data. Our findings have implications for selection testing and temporal binning in general, as well as demonstrating the usefulness of simulations for evaluating methods newly introduced to the field.

Classifying evolutionary forces in language change using neural networks

“…Note, however, that the FIT is not applicable for approximately 120 of the 1000 generated time series, since they fail the normality test. As was shown in previous analyses (Feder et al, 2014;Karjus et al, 2020), the statistical power of the FIT is low with very small selection coefficients. At β = 0.001, most simulated time series are virtually indistinguishable from stochastic drift (6% accurate and 8.5% inapplicable).…”

“…On the one hand, there are still several issues with the FIT that we have not addressed in the current study, and, on the other hand, the TSC itself is not without flaws either. We highlight two additional problems of the FIT mentioned in the literature (Karjus et al, 2020). First, the FIT assumes a constant selection coefficient β for the entire investigated period.…”

“…Interestingly, these are often cases where the frequency increments are not normally distributed, for example, the verb dream is (predominantly) classified as drift by the FIT, while the TSC marks it as subject to selection. The sharp rising frequency curve of dream in Figure 4 is probably the reason why the frequency increments are not normally distributed: as noted before (Karjus et al, 2020), the FIT does not cope well with such S-curve-like increases, with non-normally distributed frequency increments and high selection coefficients. Similarly, we can explain the differences between FIT and TSC for verbs such as spill, spoil and spell, which are also characterized by rapidly increasing, non-normally distributed frequency increments.…”

“…different constants c. The middle panel presents the results for the fixed-width binning for the seven different binning strategies of 1, 5, 10, 15, 20, 25 and 40 years. The top and middle panels are exact reproductions of the results shown in Figure 1 of Karjus et al (2020). The circles represent time series that meet the normality assumption of the frequency increments.…”

A Standardized Project Gutenberg Corpus for Statistical Analysis of Natural Language and Quantitative Linguistics

“…Analysis of natural language from a complex systems perspective has provided new insights into statistical properties of language, such as statistical laws [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ], networks [ 10 , 11 , 12 , 13 , 14 ], language change [ 15 , 16 , 17 , 18 , 19 , 20 ], quantification of information content [ 21 , 22 , 23 , 24 ], or the role of syntactic structures [ 25 ] or punctuation [ 26 ], etc. In particular, the availability of new and large publicly available datasets such as the google-ngram data [ 27 ], the full Wikipedia dataset [ 28 , 29 ], or Twitter [ 30 ] opened the door for new large-scale quantitative approaches.…”