Surprisal does not explain syntactic disambiguation difficulty: evidence from a large-scale benchmark

Abstract

Prediction has been proposed as an overarching principle that explains human information processing in language and beyond. To what degree can processing difficulty in syntactically complex sentences - one of the major concerns of psycholinguistics - be explained by predictability, as estimated using computational language models? A precise, quantitative test of this question requires a much larger scale data collection effort than has been done in the past. We present the Syntactic Ambiguity Processing Benchmark, a dataset of self-paced reading times from 2000 participants, who read a diverse set of complex English sentences. This dataset makes it possible to measure processing difficulty associated with individual syntactic constructions, and even individual sentences, precisely enough to rigorously test the predictions of computational models of language comprehension. We find that the predictions of language models with two different architectures sharply diverge from the reading time data, dramatically underpredicting processing difficulty, failing to predict relative difficulty among different syntactic ambiguous constructions, and only partially explaining item-wise variability. These findings suggest that prediction is most likely insufficient on its own to explain human syntactic processing.

Causality and signalling of garden-path sentences

“…Schijndel & Linzen [13] generated a linear regression model of reading times as a function of surprisal, and used it to predict garden-path effects. Similar to us, they observed that surprisal underestimates the garden-path effects, but different from us, they could not distinguish between the NP/S and NP/Z sentences [13,26,27]. Although statistical tests are not quoted in [13], our model clearly outperforms the results obtained from surprisal and our predictions for NP/Z sentences are significantly higher than for NP/S sentences, see table 9.…”

“…Indeed, the reading times used for doing the linear regression were only averages of reading times across different sentences and different participants; and this caused discrepancies in our results. We plan to use more detailed datasets such as a recent one released in [27]…”

“…In [25], he used the probabilities obtained from a probabilistic context-free parser to predict the existence of a garden-path effect. In the following work, empirical correlations between surprisal and self-paced reading times of garden-path sentences were also studied [13,[26][27][28]. It was shown that, although the surprisal calculated using different language models is able to predict a garden-path effect, it consistently underestimates its magnitude.…”

Deep problems with neural network models of human vision

“…We do agree with Houghton et al that it can be useful to compare language in DNNs and humans to explore the capacities of DNNs that do not have any language-specific learning mechanism. But at present, not only do the learning objectives and learning constraints seem wildly different in the two systems, but also, the performance of fully trained models "sharply diverges" from humans in controlled experiments (Huang et al, 2023).…”

Word Frequency and Predictability Dissociate in Naturalistic Reading

“…In other words, although prior evidence favors an inferential (rather than a procedural preactivation-based) interpretation of predictability effects and thus implicates inference as a key “causal bottleneck” on processing demand, the present finding of surprisal-independent frequency effects could suggest limits on the scope of this bottleneck: frequency (and thus plausibly lexical retrieval) also plays a large and surprisal-independent role in determining how long participants spend reading words. Given the remarkable success of surprisal in accounting for a range of language processing phenomena across diverse experimental measures (Demberg & Keller, 2008 ; Frank & Bod, 2011 ; Frank et al, 2015 ; Heilbron et al, 2022 ; Hoover et al, 2023 ; Lopopolo et al, 2017 ; Roark et al, 2009 ; Shain et al, 2020 , in press ; Smith & Levy, 2013 ; van Schijndel & Schuler, 2015 ; Wilcox et al, 2020 ), discoveries highlighting the explanatory limits of surprisal offer opportunities for new insights into the mechanisms and representational format of incremental meaning construction during language comprehension (e.g., Huang et al, 2023 ; van Schijndel & Linzen, 2021 ).…”