Context-Based Facilitation of Semantic Access Follows Both Logarithmic and Linear Functions of Stimulus Probability

Abstract

Stimuli are easier to process when the preceding context (e.g., a sentence, in the case of a word) makes them predictable. However, it remains unclear whether context-based facilitation arises due to predictive preactivation of a limited set of relatively probable upcoming stimuli (with facilitation then linearly related to probability) or, instead, arises because the system maintains and updates a probability distribution across all items, as posited by accounts (e.g., surprisal theory) assuming a logarithmic function between predictability and processing effort. To adjudicate between these accounts, we measured the N400 component, an index of semantic access, evoked by sentence-final words of varying probability, including unpredictable words, which are never generated in human production norms. Word predictability was measured using both cloze probabilities and a state-of-the-art machine learning language model (GPT-2). We reanalyzed five datasets (n=138) to first demonstrate and then replicate that context-based facilitation on the N400 is graded and dissociates even among words with cloze probabilities at or near 0, as a function of very small differences in model-estimated predictability. Furthermore, we established that the relationship between word predictability and context-based facilitation on the N400 is neither purely linear nor purely logarithmic but instead combines both functions. We argue that such a composite function reveals properties of the mapping between words and semantic features and how feature- and word- related information is activated during on-line processing. Overall, the results provide powerful evidence for the role of internal models in shaping how the brain apprehends incoming stimulus information.

The power of “good”: Can adjectives rapidly decrease as well as increase the availability of the upcoming noun?

“…However, the CP of adjectives is notoriously difficult to estimate using behavioral measures, as participants do not generally provide an adjective as a likely continuation of a sentence; see, for example, Boudewyn et al (2015). Here, we instead estimated the predictability of the adjectives using GPT-2 (1558M parameter version), a state-of-the-art Transformer-based neural network model of language (Radford et al, 2018; see Szewczyk & Federmeier, 2021 for validation of this technique as a proxy for CPs). We estimated the log-probability of our adjectives as continuations of the sentences and then tested whether these log-probabilities predict the amplitude of the N400 (see Figure 7).…”

“…As in the analysis of updating at the adjective (Kullback-Leibler divergence), we conducted a sensitivity analysis. A reanalysis of four experiments that used similar sentences (Szewczyk & Federmeier, 2021) showed that log(p) had a mean effect = 0.22 for target words that were mostly nouns. For adjectives, we expected a similar or a weaker effect.…”

“…First, we found N400 effects at the adjectives yoked to their cloze probability in the sentence context, as estimated by the GPT-2 model. This index captures the amount of context-based facilitation for the adjective itself (see Szewczyk & Federmeier, 2021). Second, we also found that N400 amplitudes to the adjectives varied depending on the amount of support they received from the most predictable noun.…”

“…The priors for CP effects were based on the rich literature assessing N400 effects of CP. Typically, a change in CP = 0 to CP = 1 translates to a 2–4 µV reduction of the N400 amplitude (see, e.g., Kutas & Hillyard, 1984; DeLong et al, 2005; Nieuwland et al, 2018; Szewczyk & Wodniecka, 2020; Szewczyk & Federmeier, 2021). Priors for all predictors were set such that they were also consistent with a null effect.…”

“…For the remaining predictors, we used principled priors. Priors for the baselines were based on our experience with baseline predictors (see, e.g., Szewczyk & Wodniecka, 2020; Szewczyk & Federmeier, 2021, where typically, the preword baseline for the N400 effect for data after a 0.1 Hz high-pass filter had 0.3–0.7 weight. Because the baseline weight decreases with increasing time distance from the baseline, in the analysis of the noun we set the preadjectival prior at half of the prenominal baseline.…”