Language Models Explain Word Reading Times Better Than Empirical Predictability

Abstract

Though there is a strong consensus that word length and frequency are the most important single-word features determining visual-orthographic access to the mental lexicon, there is less agreement as how to best capture syntactic and semantic factors. The traditional approach in cognitive reading research assumes that word predictability from sentence context is best captured by cloze completion probability (CCP) derived from human performance data. We review recent research suggesting that probabilistic language models provide deeper explanations for syntactic and semantic effects than CCP. Then we compare CCP with three probabilistic language models for predicting word viewing times in an English and a German eye tracking sample: (1) Symbolic n-gram models consolidate syntactic and semantic short-range relations by computing the probability of a word to occur, given two preceding words. (2) Topic models rely on subsymbolic representations to capture long-range semantic similarity by word co-occurrence counts in documents. (3) In recurrent neural networks (RNNs), the subsymbolic units are trained to predict the next word, given all preceding words in the sentences. To examine lexical retrieval, these models were used to predict single fixation durations and gaze durations to capture rapidly successful and standard lexical access, and total viewing time to capture late semantic integration. The linear item-level analyses showed greater correlations of all language models with all eye-movement measures than CCP. Then we examined non-linear relations between the different types of predictability and the reading times using generalized additive models. N-gram and RNN probabilities of the present word more consistently predicted reading performance compared with topic models or CCP. For the effects of last-word probability on current-word viewing times, we obtained the best results with n-gram models. Such count-based models seem to best capture short-range access that is still underway when the eyes move on to the subsequent word. The prediction-trained RNN models, in contrast, better predicted early preprocessing of the next word. In sum, our results demonstrate that the different language models account for differential cognitive processes during reading. We discuss these algorithmically concrete blueprints of lexical consolidation as theoretically deep explanations for human reading.

The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing

“…It is well documented that for humans, words that are less expected are harder to process-for example, during reading, people spend more time looking at words which are less predictable given context (e.g., Ehrlich and Rayner, 1981;Balota et al, 1985;McDonald and Shillcock, 2003b,a;Smith and Levy, 2013;Goodkind and Bicknell, 2018;Wilcox et al, 2020;Brothers and Kuperberg, 2021;Meister et al, 2021;Hofmann et al, 2022). We may write this general relationship as:…”

“…The relationship between surprisal and human processing time has received attention in a large number of studies Levy, 2010, 2012;Boston et al, 2008;Brothers and Kuperberg, 2021;Demberg and Keller, 2008;Fernandez Monsalve et al, 2012;Frank, 2009;Frank et al, 2013;Futrell, 2017;Futrell et al, 2020;Goodkind andBicknell, 2018, 2021;Hale, 2001;Hofmann et al, 2017Hofmann et al, , 2022Jin and Schuler, 2020;Jurafsky, 1996;Levy, 2005Levy, , 2008Levy, , 2011Levy, , 2013Levy, , 2018Lowder et al, 2018;McDonald and Shillcock, 2003a,b;Mitchell et al, 2010;Jurafsky, 2001, 2004;Rasmussen and Schuler, 2018;Reichle et al, 2003;Roark et al, 2009;van Schijndel and Linzen, 2021;Levy, 2008a,b, 2013;Wilcox et al, 2020). We will refer to literature focusing on this relationship as work on surprisal theory.…”

“…A large number of studies have simply assumed a linear relationship, explicitly-or implicitly, by the use of linear statistical models for their analysis (Reichle et al, 2003;Demberg and Keller, 2008;Boston et al, 2008;Frank, 2009;Roark et al, 2009;Mitchell et al, 2010;Fernandez Monsalve et al, 2012;Frank et al, 2013;Lowder et al, 2018;Aurnhammer and Frank, 2019;Hao et al, 2020;Merkx and Frank, 2021). A smaller number of papers, beginning with Levy (2008a, 2013), have investigated the shape of the linking function directly by making use of generalized additive models (GAMs; Wood, 2004Wood, , 2017, a family of statistical models which allows the fitting of arbitrary nonlinear relationships Levy, 2008a, 2013;Goodkind and Bicknell, 2018;Wilcox et al, 2020;Hofmann et al, 2022). For the most part, these studies have found support for the assumption of linearity.…”

“…Second, there is considerable variability between individuals in reading times and other psychometric measures of language processing (see Farmer et al, 2012). While GAMs allow the fitting an overall effect while controlling for arbitrary nonlinear by-subject effects, previous studies have either not controlled for such effects, (Smith and Levy, 2013;Wilcox et al, 2020;Hofmann et al, 2022), 5 or assumed they were just constant offsets (Goodkind and Bicknell, 2018).…”

“…However, despite the widespread recognition of these empirical facts, and the large number of studies looking at surprisal as an empirical predictor of processing time (e.g., Demberg and Keller, 2008;Levy, 2008a, 2013;Goodkind and Bicknell, 2018;Wilcox et al, 2020;Meister et al, 2021;Hofmann et al, 2022), to our knowledge no sentence processing algorithm has been proposed for which incremental runtime intrinsically increases as a function of surprisal.…”