From eye movements to scanpath networks: A method for studying individual differences in expository text reading

Ma, Xiaochuan; Liu, Yikang; Clariana, Roy B.; Gu, Chanyuan; Li, Ping

doi:10.3758/s13428-022-01842-3

Cited by 12 publications

(15 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Two fMRI datasets about sentence reading were used in this study: the Mars subset of the Reading Brain project ( 48 – 50 ) in which the sentences are connected to make a coherent story and the dataset from Pereira et al ( 1 ) in which the stimuli are dominated by unconnected relationships (see Materials and Methods for details). We refer to the two datasets as the “Reading-brain2019 dataset” and the “Pereira2018 dataset.” To identify the contributions of NSP to model-brain alignment, we trained two types of models using the BERT architecture: the MLM model that performed only the MLM task and the MLM_NSP model that performed both the MLM (for word prediction) and NSP (for sentence coherence prediction/evaluation) tasks.…”

Section: Resultsmentioning

confidence: 99%

“…At the individual difference level, our study showed that model-brain alignment computed with the MLM_NSP and MLM models was negatively correlated with reading time, suggesting that greater alignment between brains and models may be associated with faster reading. Reading time is one of the critical components for assessing reading skills ( 67 – 69 ), which has been used to differentiate skilled and less skilled readers during discourse comprehension ( 50 ). Skilled readers, compared with less skilled readers, may be more efficient in selecting and organizing key contents to construct and integrate the mental representation, thus giving rise to quicker reading time ( 50 , 70 ).…”

Section: Discussionmentioning

confidence: 99%

“…Reading time is one of the critical components for assessing reading skills ( 67 – 69 ), which has been used to differentiate skilled and less skilled readers during discourse comprehension ( 50 ). Skilled readers, compared with less skilled readers, may be more efficient in selecting and organizing key contents to construct and integrate the mental representation, thus giving rise to quicker reading time ( 50 , 70 ). Our finding of the significant correlations between model-brain alignment and reading time during discourse comprehension demonstrates that LLMs may be capable of characterizing the neurocognitive map of skilled comprehension.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Yu,

Gu,

Huang

et al. 2024

Sci. Adv.

Self Cite

View full text Add to dashboard Cite

Current large language models (LLMs) rely on word prediction as their backbone pretraining task. Although word prediction is an important mechanism underlying language processing, human language comprehension occurs at multiple levels, involving the integration of words and sentences to achieve a full understanding of discourse. This study models language comprehension by using the next sentence prediction (NSP) task to investigate mechanisms of discourse-level comprehension. We show that NSP pretraining enhanced a model’s alignment with brain data especially in the right hemisphere and in the multiple demand network, highlighting the contributions of nonclassical language regions to high-level language understanding. Our results also suggest that NSP can enable the model to better capture human comprehension performance and to better encode contextual information. Our study demonstrates that the inclusion of diverse learning objectives in a model leads to more human-like representations, and investigating the neurocognitive plausibility of pretraining tasks in LLMs can shed light on outstanding questions in language neuroscience.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Yu,

Gu,

Huang

et al. 2024

Sci. Adv.

Self Cite

View full text Add to dashboard Cite

show abstract

“…According to the "eye-mind" hypothesis [31], eye movement is related to a user's attention being directed by the interface. By de ning areas of interest on certain parts of the system interface [32], combined with sequence of xations, researchers can analyze the paths of the eye movements and derive meaningful information [33][34][35][36]. Therefore, this study employed an eye tracker to collect eye movement data and use it to nd users' search tactics and strategies with metadata.…”

Section: Research Backgroundmentioning

confidence: 99%

Using eye movement sequences and hidden Markov model to explore strategy blending in metadata-supported image search

Lin

Chan

2023

Preprint

View full text Add to dashboard Cite

This study explores users’ search strategies associated with different information aids in an image search context. We investigate "strategy blending", i.e., the mixture of different strategies (tag-based and header-based) in a display with both possibilities. Using eye-movement-recordings, supported by Hidden Markov Model (HMM) modeling, we attempt to reveal strategies and tactics as well as the blending of different strategies. The findings make theoretical contributions to the literature on strategy blending and information seeking behavior and provide practitioners with guidelines on metadata support for website design to enhance the user experience and meet user needs. In our understanding of this domain, we are the first to bridge strategies in decision making to search strategies with actual users instead of using mere simulation. Strategy blending can be explained by investigating user search behavior in an image search context. HMMs can be used to discover latent search tactics, and user eye movement sequences reveal different types of strategy blending. For practical design in the context of image search, metadata is indeed useful to assist image search as a navigation support, and represents a better chance to fulfill the information needs of users. Responding to the emerging digital environment and the new paradigm of people’s search behavior with various devices, these results can be generalized to other research fields, such as mobile system design or user modeling, to satisfy various users with different needs.

show abstract

“…In the case of silent reading, gazing at one spot for a long time occurs for words and characters that are difficult to read (difficult-to-read characters) [1,2]. Therefore, gaze may be used to automatically detect difficult-to-read characters.…”

Section: Introductionmentioning

confidence: 99%

Automated Correction of Eye‐Tracking Data for Detecting Difficult‐to‐Read Characters in Japanese Text

Ito,

Ishizawa,

Kageyama

2024

IEEJ Transactions Elec Engng

View full text Add to dashboard Cite

This paper proposes a new eye‐tracking data correction method for the development of an automatic detection system for difficult‐to‐read characters in Japanese text. The proposed method, which adjusts the x‐axis of the eye, was compared with existing methods that adjust only the y‐axis. Data correction was performed for each method, and the detection accuracy of difficult‐to‐read Japanese characters was calculated and compared. The experimental results indicate that adjusting the x‐axis and y‐axis can improve the detection accuracy of difficult‐to‐read characters in Japanese text. This research advances easy correction techniques for difficult‐to‐read character detection in Japanese text and broadens the use of eye trackers, including low‐cost eye trackers. © 2024 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

show abstract

From eye movements to scanpath networks: A method for studying individual differences in expository text reading

Cited by 12 publications

References 95 publications

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension

Using eye movement sequences and hidden Markov model to explore strategy blending in metadata-supported image search

Automated Correction of Eye‐Tracking Data for Detecting Difficult‐to‐Read Characters in Japanese Text

Contact Info

Product

Resources

About