Predicting Question Difficulty in Web Surveys: A Machine Learning Approach Based on Mouse Movement Features

Abstract

Survey research aims to collect robust and reliable data from respondents. However, despite researchers’ efforts in designing questionnaires, survey instruments may be imperfect, and question structure not as clear as could be, thus creating a burden for respondents. If it were possible to detect such problems, this knowledge could be used to predict problems in a questionnaire during pretesting, inform real-time interventions through responsive questionnaire design, or to indicate and correct measurement error after the fact. Previous research has used paradata, specifically response times, to detect difficulties and help improve user experience and data quality. Today, richer data sources are available, for example, movements respondents make with their mouse, as an additional detailed indicator for the respondent–survey interaction. This article uses machine learning techniques to explore the predictive value of mouse-tracking data regarding a question’s difficulty. We use data from a survey on respondents’ employment history and demographic information, in which we experimentally manipulate the difficulty of several questions. Using measures derived from mouse movements, we predict whether respondents have answered the easy or difficult version of a question, using and comparing several state-of-the-art supervised learning methods. We have also developed a personalization method that adjusts for respondents’ baseline mouse behavior and evaluate its performance. For all three manipulated survey questions, we find that including the full set of mouse movement measures and accounting for individual differences in these measures improve prediction performance over response-time-only models.

Classification ensembles for multivariate functional data with application to mouse movements in web surveys

“…In CAI and web survey modes, for example, response times have been one of the most popular paradata types considered for predicting break-offs in web surveys and designing interventions (Mittereder and West, 2021), studying the relationship of response times and measurement errors (Heerwegh, 2003) or detecting sources of difficulty in the survey (Conrad et al, 2007;Yan and Tourangeau, 2008). However, response times are not always reliable descriptors of the entire survey process (e.g., a long response time may not necessarily be due to the survey question but to other nonsurvey-related tasks such as checking or responding to emails, see Horwitz et al, 2017;Fernández-Fontelo et al, 2021). Web surveys, in particular, provide another promising source of paradata in the form of computer mouse movements, which have recently demonstrated the ability to convey additional information beyond response times (O'Hora et al, 2016;Stillman et al, 2017;Horwitz et al, 2017Horwitz et al, , 2020Fernández-Fontelo et al, 2021) Mouse movements have been used so far in a number of applications in different fields (Freeman, 2018;Chen et al, 2001).…”

“…However, response times are not always reliable descriptors of the entire survey process (e.g., a long response time may not necessarily be due to the survey question but to other nonsurvey-related tasks such as checking or responding to emails, see Horwitz et al, 2017;Fernández-Fontelo et al, 2021). Web surveys, in particular, provide another promising source of paradata in the form of computer mouse movements, which have recently demonstrated the ability to convey additional information beyond response times (O'Hora et al, 2016;Stillman et al, 2017;Horwitz et al, 2017Horwitz et al, , 2020Fernández-Fontelo et al, 2021) Mouse movements have been used so far in a number of applications in different fields (Freeman, 2018;Chen et al, 2001). In survey research, mouse movement measures (i.e., features of mouse movement patterns such as the number of changes in direction or the number of times a participant is inactive for a certain period of time) have also shown promising results: Stieger and Reips (2010), for example, found that several mouse movement measures in an online questionnaire (e.g., longer inactivities or an excessive number of clicks) had a negative correlation to data quality, and Horwitz et al (2017) demonstrated -in the controlled environment of a laboratorythat mouse movement measures other than response times were good predictors of respondents' perceived difficulty with an online survey question.…”

“…In survey research, mouse movement measures (i.e., features of mouse movement patterns such as the number of changes in direction or the number of times a participant is inactive for a certain period of time) have also shown promising results: Stieger and Reips (2010), for example, found that several mouse movement measures in an online questionnaire (e.g., longer inactivities or an excessive number of clicks) had a negative correlation to data quality, and Horwitz et al (2017) demonstrated -in the controlled environment of a laboratorythat mouse movement measures other than response times were good predictors of respondents' perceived difficulty with an online survey question. Recent studies in the field showed that several mouse movements reflected issues with items of a web survey (Horwitz et al, 2020) and further that these measures were also good predictors of such issues (Fernández-Fontelo et al, 2021). Although all previous papers have demonstrated the potential of a number of mouse movement measures as scalar-valued features to improve some survey aspects, none of them have yet considered using the entire mouse movement trajectory as a bivariate function of the mouse cursor positions on the computer screen over time, which is the most detailed representation of the available information.…”

“…Here, difficulty is of interest as it correlates with measurement errors in responses and thus worse survey data quality (Yan and Olson, 2013). To this end, our application is based on a web survey that contained a number of questions (henceforth "target questions") that were experimentally manipulated to create two different levels of difficulty; for each of these target questions, mouse movements were collected (more details are given in Section 3; see also Horwitz et al, 2020;Fernández-Fontelo et al, 2021). The purpose of the application is thus to find a predictive model using bivariate mouse movement trajectories as functional predictors to classify easy and difficult question settings.…”

“…Then, typical machine learning methods for classification tasks may be considered using the functional features as multivariate predictors. In the context of survey research, Fernández-Fontelo et al (2021) used scalar-valued features of (bivariate) mouse movement trajectories as multivariate predictors in a number of machine learning models to predict question difficulty in a web survey successfully.…”