CMCL 2021 Shared Task on Eye-Tracking Prediction

Abstract

Eye-tracking data from reading represent an important resource for both linguistics and natural language processing. The ability to accurately model gaze features is crucial to advance our understanding of language processing. This paper describes the Shared Task on Eye-Tracking Data Prediction, jointly organized with the eleventh edition of the Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2021). The goal of the task is to predict 5 different token-level eyetracking metrics from the Zurich Cognitive Language Processing Corpus (ZuCo). Eyetracking data were recorded during natural reading of English sentences. In total, we received submissions from 13 registered teams, whose systems include boosting algorithms with handcrafted features, neural models leveraging transformer language models, or hybrid approaches. The winning system used a range of linguistic and psychometric features in a gradient boosting framework.

The ZuCo Benchmark on Cross-Subject Reading Task Classification with EEG and Eye-Tracking Data

“…Additionally, adding tasks such as eye movement and ERP prediction would be beneficial for various research communities. For example, the prediction of eye movement patterns has gained interest also in the NLP community (Hollenstein et al, 2021a). The main goal of this work is to create a platform for discussion and future research on a common benchmark task for reading task classification based on eye movement and brain activity data.…”

“…The ZuCo dataset is publicly available and has recently been used in a variety of applications including leveraging EEG and eye-tracking data to improve NLP tasks (Barrett et al, 2018; Mathias et al, 2020; McGuire and Tomuro, 2021), evaluating the cognitive plausibility of computational language models (Hollenstein et al, 2019b; Hollenstein and Beinborn, 2021), investigating the neural dynamics of reading (Pfeiffer et al, 2020), developing models of human reading (Bautista and Naval, 2020; Bestgen, 2021). Recently, ZuCo has also been leveraged for an ML competition on eye-tracking prediction (Hollenstein et al, 2021a). This shows that the ZuCo dataset has been used successfully for a wide range of ML tasks.…”

On the Trade-off between Redundancy and Local Coherence in Summarization

“…On the one hand, recent work in cognitive psychology -the field that aims to answer questions about how humans think-employed natural language processing (NLP) models to investigate aspects of human language comprehension. For instance, language models and statistical parsers were used to explain language processing difficulty (Sarti et al, 2021;Rathi, 2021;Meister et al, 2022) and incrementality (Merkx & Frank, 2021;Stanojević et al, 2021), syntactic agreement processes (Ryu & Lewis, 2021), brain representations of abstract and concrete concepts (Anderson et al, 2017;Ramakrishnan & Deniz, 2021), prediction of gaze behaviour (Hollenstein et al, 2021), among others. In clinical psychology, Transformer-based language models are being used to formulate cognitive models that better explain human emotions (Guo & Choi, 2021), comprehension deficit in aphasia subjects (Guo & Choi, 2021), and even improve suicidal prevention systems (MacAvaney et al, 2021).…”

LAST at CMCL 2021 Shared Task: Predicting Gaze Data During Reading with a Gradient Boosting Decision Tree Approach

“…The task was to predict five eye-tracking features, averaged across all participants and scaled in the range between 0 and 100, for each word of a series of sentences: (1) the total number of fixations (nFix), (2) the duration of the first fixation (FFD), (3) the sum of all fixation durations, including regressions (TRT), (4) the sum of the duration of all fixations prior to progressing to the right, including regressions to previous words (GPT), and (5) the proportion of participants that fixated the word (fixProp). These dependent variables (DVs) are described in detail in Hollenstein et al (2021). The submissions were evaluated using the mean absolute error (MAE) metric and the systems were ranked according to the average MAE across all five DVs, the lowest being the best.…”