Analyzing cognitive processes in CATI-surveys with response latencies : an empirical evaluation of the consequences of using different baseline speed measures

Mayerl, Jochen; Sellke, Piet; Urban, Dieter

doi:10.18419/opus-5466

Cited by 3 publications

(1 citation statement)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, response time is a relatively coarse measure that does not specify what might have caused any observed latency: Respondents who take a long time to answer may not even be engaged in the survey task, but checking email, talking on the phone or tending to a child (Höhne & Schlosser, 2018;Sendelbah et al, 2016). Also, direct comparisons of absolute response times between participants may be problematic because each person has his or her own baseline speed (Mayerl et al, 2005), and models for response times that include task characteristics and respondent features are still in their infancy (Couper & Kreuter, 2013).…”

Section: Introductionmentioning

confidence: 99%

Predicting Question Difficulty in Web Surveys: A Machine Learning Approach Based on Mouse Movement Features

Fernández‐Fontelo

Kieslich

Henninger

et al. 2021

Social Science Computer Review

View full text Add to dashboard Cite

Survey research aims to collect robust and reliable data from respondents. However, despite researchers’ efforts in designing questionnaires, survey instruments may be imperfect, and question structure not as clear as could be, thus creating a burden for respondents. If it were possible to detect such problems, this knowledge could be used to predict problems in a questionnaire during pretesting, inform real-time interventions through responsive questionnaire design, or to indicate and correct measurement error after the fact. Previous research has used paradata, specifically response times, to detect difficulties and help improve user experience and data quality. Today, richer data sources are available, for example, movements respondents make with their mouse, as an additional detailed indicator for the respondent–survey interaction. This article uses machine learning techniques to explore the predictive value of mouse-tracking data regarding a question’s difficulty. We use data from a survey on respondents’ employment history and demographic information, in which we experimentally manipulate the difficulty of several questions. Using measures derived from mouse movements, we predict whether respondents have answered the easy or difficult version of a question, using and comparing several state-of-the-art supervised learning methods. We have also developed a personalization method that adjusts for respondents’ baseline mouse behavior and evaluate its performance. For all three manipulated survey questions, we find that including the full set of mouse movement measures and accounting for individual differences in these measures improve prediction performance over response-time-only models.

show abstract