It is all in the surv-eye: can eye tracking data shed light on the internal consistency in self-report questionnaires on cognitive processing strategies?

Abstract

Although self-report questionnaires are widely used, researchers debate whether responses to these types of questionnaires are valid representations of the respondent's actual thoughts and beliefs. In order to provide more insight into the quality of questionnaire data, we aimed to gain an understanding of the processes that impact the completion of self-report questionnaires. To this end, we explored the process of completing a questionnaire by monitoring the eye tracking data of 70 students in higher education. Specifically, we examined the relation between eye movement measurements and the level of internal consistency demonstrated in the responses to the questionnaire. The results indicated that respondents who look longer at an item do not necessarily have more consistent answering behaviour than respondents with shorter processing times. Our findings indicate that eye tracking serves as a promising tool to gain more insight into the process of completing self-report questionnaires.

Tracking Patterns in Self-Regulated Learning Using Students’ Self-Reports and Online Trace Data

“…This is particularly pressing since existing research suggests that learners tend to use ineffective learning strategies (Jamieson-Noel & Winne, 2003), and do not make effective use of available resources to optimize their learning, even in those environments that build on effective learning designs (Ellis, et al, 2005;Lust, et al, 2013). As discussed in this special issue, potential reasons are students' cognitive processing capacity (Chauliac, et al, 2020), exerted effort (Iaconelli & Wolters, 2020), and student characteristics (Vriesema & McCaslin, 2020). Comparing self-reported data with online traces of learning behaviour taps right into a potential upward bias in self-reports on motivation and strategy use.…”

Self-Report is Indispensable to Assess Students’ Learning

“…Finally, in terms of developing additional methods for testing validity, Chauliac et al (2020) observed university students' gaze behavior while answering items on a questionnaire assessing habitual use of different cognitive strategies during learning from texts. There were systematic links between number and duration of fixations, on the one hand, and the consistency of answering different items from the same scale, on the other.…”

A Proposed Remedy for Grievances about Self-Report Methodologies

“…Only one study used think aloud procedures. Chauliac, et al (2020) asked participants to rate how frequently they applied various cognitive processes while studying textual information. Durik and Jenkins (2020) administered survey items calling for respondents to the degree to which they agreed with statements describing interest in several areas of study: astronomy, biology, math and psychology; and a single item inviting respondents to classify how certain they were about the set of ratings of interest.…”

“…Examples are an experience in the immediate past, such as a just-completed session of collaborative work or a class period that has just finished. Moeller and colleagues (2020) minimized the time interval with their experience sampling method, as did Chauliac et al (2020) by administering their survey after participants had just completed a studying task. The other end of this spectrum is a generalized setting, such as studying or life in school.…”

“…Second, it is often the case that self-report data are among other predictors used to predict an outcome. Multiple regression and similar models are common statistical methods applied in this case, as was true for studies in this special issue (Chauliac et al, 2020;Durik & Jenkins, 2020;van Halem et al, 2020;Moeller et al, 2020;Vriesema & McCaslin, 2020). In these types of analysis, each predictor is residualized for every other predictor.…”