A Typology of Web Survey Paradata for Assessing Total Survey Error

Abstract

This article reviews the existing literature on the collection of paradata in web surveys and extends the research in this area beyond the commonly studied measurement error problem to paradata that can be collected for managing and mitigating other important sources of error. To do so, and in keeping with the nature of paradata as process-oriented, we develop a typology of web survey paradata that incorporates information from all steps in the web survey process. We first define web survey paradata and describe general phases of paradata that run parallel to the steps in fielding a typical web survey. Within each phase, we enumerate several errors within the total survey error paradigm that can be examined with paradata, discussing case studies and motivating examples that illustrate innovative uses of paradata across the web survey process. We conclude with a discussion of open questions and opportunities for further work in this area. Overall, we develop this typology keeping technological advancements at the center of our discussion, but with flexibility to continuously incorporate new developments and trends in both technology and study design. Our typology encourages researchers to think about paradata as tools that can be used to investigate a broader range of outcomes than previously studied.

Understanding Respondents’ Attitudes Toward Web Paradata Use

“…Questionnaire navigation paradata (e.g., time stamps, mouse clicks, and loss of focus) can be used to reconstruct and describe the questionnaire completion process and allow for conclusions to be drawn about the survey-taking experience. Both device-type and questionnaire navigation paradata can be used to evaluate and improve survey instruments, to better understand how respondents answer questions, and, ultimately, to improve the quality of measurement in web surveys (Callegaro, 2013; McClain et al, 2019).…”

How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items

“…Callegaro (2012) differentiates paradata describing the device type and paradata describing the questionnaire navigation. The typology of McClain et al (2018) describes four phases (prior survey, recruitment, access, and response). We combine both views by reorganizing paradata into three main categories of access-related, response-related and process-related paradata, 2 partitioned in additional sub-categories (presented in Table 1 with prototypical examples).…”

“…In the following, we will first refer to taxonomies and typologies of paradata (Callegaro 2012;Olson and Parkhurst 2013;McClain et al 2018) showing that paradata and log data can be grouped on a surface level into three generic categories of access-related, response-related and process-related paradata. The taxonomy is not intended to be an exhaustive literature review, but rather an overview of examples that constitute the essence for categories and subcategories of paradata.…”

“…The aggregated information from previous assessments classified byMcClain et al (2018) as prior survey phase represents auxiliary data, which can be linked if available. Besides, we disentangle responserelated and process-related paradata to express the proximity of paradata dealing directly with the response and paradata dealing with the navigation.…”

Informed consent for paradata use in web surveys

“…In web surveys, paradata are usually collected automatically as a by-product of computer-assisted data collection. They include, for example, the device and browser that respondents used to fill out a questionnaire, the way they navigate through questions by going back and forth, changing given answers, interrupting the survey for a certain time, and so on (Callegaro, 2013; McClain et al, 2019).…”