Randomization tests in language typology

Abstract

Two of the major assumptions that common statistical tests make about random The limits of classical statistical tests in typologyTwo major assumptions that common statistical tests rely upon are not tenable for most typological data. We suggest to use randomization tests, which avoid these assumptions and which are applicable to a wide range of data types, so most typological data can be analyzed with the same tools. To establish our case for the use of randomization-based tests, we will first discuss the underpinnings of the two types of statistical tests that are most often used by typologists, and by social scientists in general. The two types of tests are the parametric tests (such as the t-test) and the classical non-parametric tests (such as the chi-square test). Both types of tests make assumptions which can be problematic for typological data. We will focus on the assumptions of the parametric tests first.

Twice random, once mixed: Applying mixed models to simultaneously analyze random effects of language and participants

“…suggested using a parametric bootstrap (cf. Janssen, Bickel & Zúñiga 2006) to correct the p-values of the LRT when testing for the inclusion of random effects, and this procedure has the advantage over the solution suggested by Baayen et al (2008) of not depending on a Bayesian framework.…”

Statistical modeling of language universals

“…A crude but still workable method for estimating biases is what I call the "Family Bias Method". In this method, each family is evaluated as to whether its daughter languages show a bias towards the same structure (or set of structures), as revealed by a randomization test (Janssen et al 2006). If there is such a bias (many daughter languages with the same structure), this means that daughter languages have preferentially innovated in the same direction or kept what was already in the proto-language.…”

“…All p-values are based on an asymptotic χ 2 -distribution but are in the same ballpark of what one obtains through resampling (bootstrap) tests (following the recommendation of Janssen et al 2006, with 2,000 simulations).…”

Typology in the 21st century: Major current developments

“…But exhaustive sampling makes classical statistical methods mathematically meaningless. In response to these problems, typologists are now adopting MonteCarlo and exact methods, and first steps have also been undertaken towards randomization-based reliability tests on coding (Janssen et al 2006). Unlike classical methods, which presuppose random sampling from a normally distributed population, these methods do not support statistical inference to an underlying population of all human languages.…”