<i>N</i>-gram probability effects in a cloze task

Shaoul, Cyrus; Baayen, R. Harald; Westbury, Chris

doi:10.1075/ml.9.3.04sha

Cited by 16 publications

(11 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This might in part explain some inconsistency between our SRC and PSC samples. We suggest that future studies should more closely evaluate which type of information is contained in which particular CCP sample, in order to obtain a scientifically deeper explanation for the portions of the variance that can presently still be better accounted for by CCP (Shaoul et al, 2014;Luke and Christianson, 2016;Hofmann et al, 2017;Lopukhina et al, 2021). Table 3 shows that the correlations of n-gram and RNN models with the CCP data are larger than the correlations with topics models in both data samples.…”

Section: Ccp Effects Set a Challenge For Unexplained Predictive Proce...mentioning

confidence: 97%

Language Models Explain Word Reading Times Better Than Empirical Predictability

Hofmann

Remus

Biemann

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Though there is a strong consensus that word length and frequency are the most important single-word features determining visual-orthographic access to the mental lexicon, there is less agreement as how to best capture syntactic and semantic factors. The traditional approach in cognitive reading research assumes that word predictability from sentence context is best captured by cloze completion probability (CCP) derived from human performance data. We review recent research suggesting that probabilistic language models provide deeper explanations for syntactic and semantic effects than CCP. Then we compare CCP with three probabilistic language models for predicting word viewing times in an English and a German eye tracking sample: (1) Symbolic n-gram models consolidate syntactic and semantic short-range relations by computing the probability of a word to occur, given two preceding words. (2) Topic models rely on subsymbolic representations to capture long-range semantic similarity by word co-occurrence counts in documents. (3) In recurrent neural networks (RNNs), the subsymbolic units are trained to predict the next word, given all preceding words in the sentences. To examine lexical retrieval, these models were used to predict single fixation durations and gaze durations to capture rapidly successful and standard lexical access, and total viewing time to capture late semantic integration. The linear item-level analyses showed greater correlations of all language models with all eye-movement measures than CCP. Then we examined non-linear relations between the different types of predictability and the reading times using generalized additive models. N-gram and RNN probabilities of the present word more consistently predicted reading performance compared with topic models or CCP. For the effects of last-word probability on current-word viewing times, we obtained the best results with n-gram models. Such count-based models seem to best capture short-range access that is still underway when the eyes move on to the subsequent word. The prediction-trained RNN models, in contrast, better predicted early preprocessing of the next word. In sum, our results demonstrate that the different language models account for differential cognitive processes during reading. We discuss these algorithmically concrete blueprints of lexical consolidation as theoretically deep explanations for human reading.

show abstract

Section: Ccp Effects Set a Challenge For Unexplained Predictive Proce...mentioning

confidence: 97%

Language Models Explain Word Reading Times Better Than Empirical Predictability

Hofmann

Remus

Biemann

et al. 2022

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…In contrast, EOR's participants are cued with a pattern with little semantic information and have to select a verb (that is, a form and a meaning at the same time) that fits the pattern. In this capacity, the task is similar to other psycholinguistic tasks often used for studying human memory, implicit knowledge of words, and mental grammar: the fill-in-the-blank (cloze) task, the free word association task, and the cued recall task (see Shaoul, Baayen, & Westbury, 2014, for a review).…”

Section: Theoretical Overviewmentioning

confidence: 99%

Modelling verb selection within argument structure constructions

Matusevych

Alishahi

Backus

2016

Language, Cognition and Neuroscience

View full text Add to dashboard Cite

This article looks into the nature of cognitive associations between verbs and argument structure constructions (ASCs). Existing research has shown that distributional and semantic factors affect speakers' choice of verbs in ASCs. A formal account of this theory has been proposed by Ellis, N. C., O'Donnell, M. B., & Römer, U. [(2014a). The processing of verb-argument constructions is sensitive to form, function, frequency, contingency and prototypicality.

show abstract

“…In a first step, these three LMs were used to predict CCP data (cf. Shaoul, Baayen & Westbury, 2015). Together with a baseline of word position and frequency, an n-gram model reproducibly accounted for nearly half of the itemlevel variance of the CCP data from the PSC (Dambacher & Kliegl, 2007;Kliegl et al, 2004), and therefore comes close to the best models of single-word recognition (e.g.…”

Section: The Present Studymentioning

confidence: 62%

Language models explain word reading times better than empirical predictability

Mj¹,

Remus²,

Biemann³

et al. 2020

Preprint

View full text Add to dashboard Cite

While word predictability from sentence context is typically investigated by cloze completion probabilities (CCP), it can be more deeply understood by relying on language models (LMs), allowing to define the three key components of memory: Memory starts with experience as implemented by a text corpus, here defined by Wikipedia capturing general knowledge and (movie) subtitles approximating social interactions. LMs then consolidate a long-term memory structure from experience, as addressed by n-gram, topics and recurrent neural network (RNN) models. Retrieval was investigated by predicting fixation durations from an English and a German reading sample. Item-level regressions showed greater correlations of LMs with single-fixation duration (SFD), gaze duration (GD) and total viewing time (TVT) than CCP. When predicting each fixation case separately using generalized additive models, three LMs together always performed better than CCP. When testing single LMs against the typically-sized English CCP sample (N = 30), LMs usually performed better than CCP (8 vs. 3). The larger German CCP sample (N = 272), however, often performed better than single LMs (4 vs. 2). Subtitles-trained n-gram probabilities of present (and last) words allowed for reliable predictions of all fixation durations. Wikipedia-trained topic probabilities of the last and present word allow for reliable predictions of late GD and TVT effects. The present word predictions of RNNs were less sensitive to training-corpus choice and are recommendable if a single LM is used. Moreover, its reliable next word probability effects make it most suitable to address parafoveal preview and top-down predictions.

show abstract

N-gram probability effects in a cloze task

Cited by 16 publications

References 54 publications

Language Models Explain Word Reading Times Better Than Empirical Predictability

Language Models Explain Word Reading Times Better Than Empirical Predictability

Modelling verb selection within argument structure constructions

Language models explain word reading times better than empirical predictability

Contact Info

Product

Resources

About