Observing response processes with eye tracking in international large-scale assessments: evidence from the OECD PIAAC assessment

Abstract

This paper reports on a pilot study that used eye tracking techniques to make detailed observations of item response processes in the OECD Programme for the International Assessment of Adult Competencies (PIAAC). The lab-based study also recorded physiological responses using measures of pupil diameter and electrodermal activity. The study tested 14 adult respondents as they individually completed the PIAAC computer-based assessment. The eye tracking observations help to fill an 'explanatory gap' by providing data on variation in item response processes that are not captured by other sources of process data such as think aloud protocols or computer-generated log files. The data on fixations and saccades provided detailed information on test item response strategies, enabling profiling of respondent engagement and response processes associated with successful performance. Much of that activity does not include the use of the keyboard and mouse, and involves 'off-screen' use of pen and paper (and calculator) that are not captured by assessment log-files. In conclusion, this paper points toward an important application of eye tracking in large-scale assessments. This includes insights into response processes in new domains such as adaptive problem-solving that aim to identify individuals' ability to select and combine resources from the digital and physical environment.

Undergraduate Students’ Critical Online Reasoning—Process Mining Analysis

“…A similar question needs to be researched on the basis of the available findings: can ideal solution patterns be found in (partial) sequences, and can instructional settings be developed based on these patterns, for instance, by indicating to the learner that a text should be read first, and the web search carried out subsequently and with a certain term specification? Additional explanations by the test takers, for instance, through concurrent verbal protocols, could provide further insights into the students' causal decision contexts (Leighton and Gierl, 2007) and be combined and evaluated in parallel to the time-sequential recordings of the ET data (Maddox et al, 2018) and predefined process steps.…”

Active photonic sensing for super-resolved reading performance in simulated prosthetic vision

“…Analyzing eye movement during reading can serve for studying various cognitive processing, as was demonstrated by Raney et al [58]. Data on saccades and fixation can aid in detecting patterns of readers engagement and performance [59] and to further categorize reading skills in children [60] and adults [61]. In the current study, we used words rather than sentences to measure the reading performance of our subject.…”

The risk–return trade‐off: Performance assessments and cognitive validation of inferences

“…Aside from interviews, eye tracking can also be used to collect response process data. In particular, participants’ gazes can be traced for where they are visually attending to specific material and resources and/or considering specific knowledge and skills in their formulation of a solution (Lai et al ., ; Leighton, ; Maddox et al ., ). Although eye tracking is a non‐intrusive, observational method for collecting evidence of the visual information to which individuals are attending in the process of formulating their solutions, it does not provide evidence of participants’ explicit interpretations of assessment stimuli.…”