The cognitive plausibility of statistical classification models: Comparing textual and behavioral evidence

Abstract

(100-200 words)Usage-based linguistics abounds with studies that use statistical classification models to analyse either textual corpus data or behavioral experimental data. Yet, before we can draw conclusions from statistical models of empirical data that we can feed back into cognitive linguistic theory, we need to assess whether the text-based models are cognitively plausible and whether the behavior-based models are linguistically accurate. In this paper, we review four case studies that evaluate statistical classification models of richly annotated linguistic data by explicitly comparing the performance of a corpus-based model to the behavior of native speakers. The data come from four different languages (Arabic, English, Estonian, and Russian) and pertain to both lexical as well as syntactic near-synonymy. We show that behavioral evidence is needed in order to fine-tune and improve statistical models built on data from a corpus. We argue that methodological pluralism and triangulation are the keys for a cognitively realistic linguistic theory.

General introduction: A comparative perspective on probabilistic variation in grammar

“…Second, this probabilistic knowledge is derived in large part from language experience, and so is subtly, but dynamically (re)constructed throughout speakers' lives. The probabilistic nature of grammar is supported by evidence showing that the likelihood of finding a particular linguistic variant in a particular context in a corpus corresponds to the intuitions that speakers have about the acceptability of the variants (see Bresnan & Ford 2010;Klavan & Divjak 2016). Bresnan (2007: 76-84), for example, used a scalar rating task based on corpus materials (transcriptions of spoken dialogue passages) as stimuli to model subjects' responses regarding the naturalness of dative variants in context.…”

Are corpus-based predictions mirrored in the preferential choices and ratings of native speakers? Predicting the alternation between the Estonian adessive case and the adposition <i>peal</i> ‘on’

“…One prolific area pertains to the discussion of how corpus-based frequency estimates relate to experimental findings, especially acceptability judgements (see Divjak 2016 for a recent overview). There has also been an exponential growth in published studies that use probabilistic statistical classification models to analyse linguistic data; see Klavan and Divjak (2016) for an overview. Still, only a small number of these studies have compared their findings with behavioral data (Roland et al 2006, Wasow andArnold 2003).…”

“…Klavan and, that without behavioral data it would be very difficult if not impossible to provide an adequate assessment of a corpus-based model. Linguistic experiments are necessary to calibrate our models -sometimes models are very accurate, and sometimes they appear to be less accurate; in order to set "upper and lower boundaries to what could be psychologically relevant" we need behavioral data to evaluate the corpus-based model (Klavan and Divjak 2016).…”

Towards cognitively plausible data science in language research

“…Third, although regression models have produced classification results that have received support from behavioural studies (for an overview of this relatively recent trend in linguistics see Klavan & Divjak, 2016), the algorithms these models rely on are not based on learning mechanisms but maximize likelihood using optimization techniques. Whether humans do or do not exhibit (near-)optimal behaviour remains a matter of debate (see Kahneman & Tversky, 1984).…”