An interpretable measure of semantic similarity for predicting eye movements in reading

Abstract

Predictions about upcoming content play an important role during language comprehension and processing. Semantic similarity as a metric has been used to predict how words are processed in context in language comprehension and processing tasks. This study proposes a novel, dynamic approach for computing contextual semantic similarity, evaluates the extent to which the semantic similarity measures computed using this approach can predict fixation durations in reading tasks recorded in a corpus of eye-tracking data, and compares the performance of these measures to that of semantic similarity measures computed using the cosine and Euclidean methods. Our results reveal that the semantic similarity measures generated by our approach are significantly predictive of fixation durations on reading and outperform those generated by the two existing approaches. The findings of this study contribute to a better understanding of how humans process words in context and make predictions in language comprehension and processing. The effective and interpretable approach to computing contextual semantic similarity proposed in this study can also facilitate further explorations of other experimental data on language comprehension and processing.

Optimizing Predictive Metrics for Human Reading Behavior

“…The current study’s second contribution highlights the usefulness of computational metrics in predicting language processing and comprehension across multiple languages, not only English. Our findings support previous research that demonstrates the predictability of contextual semantic relevance in eye movements during reading and comprehension (e.g., Roland et al, 2012; Frank and Willems, 2017; Sun et al, 2023a). We also found that surprisal effects were present in multiple language processing, which actually supports numerous studies using surprisal to predict language processing (Hale et al, 2022; Hale, 2016; Schrimpf et al, 2021).…”

“…Moreover, contextual semantic information can be linked with short-term memory, a notion supported by empirical investigations into memory systems like the visuospatial sketchpad and phonological loop (Baddeley, 2000;Postle, 2006;Baddeley, 2010). More holistic methods for computing contextual semantic relevance have been proposed (Sun et al, 2023a;Sun et al, 2023b;Sun, 2023). However, in terms of human forgetting mechanism (Murre and Dros, 2015), the current study proposes a method to compute contextual semantic relevance by considering the weights which are determined by the position distance between the target word and its surrounding words (Fig.…”

An interpretable semantic similarity for predicting eye movements in reading: Approach comparison

“…However, such approaches may be improved in terms of comprehensiveness in the types of neighboring words considered and/or the linguistic interpretability of the semantic similarity values obtained. Sun et al (2023a) proposed a dynamic approach for computing contextual semantic similarity, which is cognitively and linguistically interpretable. Meanwhile, Sun et al (2023a) modified the cosine (Frank and Willems, 2017) and Euclidean methods (Broderick et al, 2019), and enabled them more interpretable and effective.…”

“…Sun et al (2023a) proposed a dynamic approach for computing contextual semantic similarity, which is cognitively and linguistically interpretable. Meanwhile, Sun et al (2023a) modified the cosine (Frank and Willems, 2017) and Euclidean methods (Broderick et al, 2019), and enabled them more interpretable and effective. However, the method of Sun et al (2023a) ignored the different contribution of contextual words and the expectation effect.…”

“…Meanwhile, Sun et al (2023a) modified the cosine (Frank and Willems, 2017) and Euclidean methods (Broderick et al, 2019), and enabled them more interpretable and effective. However, the method of Sun et al (2023a) ignored the different contribution of contextual words and the expectation effect. Based on the approach of Sun et al (2023a) and inspired by the attention mechanism of Transformer (Vaswani et al, 2017), the current study proposes a "attention-aware" method to compute more powerful and interpretable semantic relevance which can incorporate the contextual information more comprehensively.…”