Experimental Evidence on Immediate and Long-term Consequences of Test-induced Respondent Burden for Panel Attrition

Abstract

Panel attrition is a major problem in long-term panel studies. While the design of the German National Educational Panel Study adult surveycombining biannual competency tests with regular face-to-face interviewsis highly innovative, such a design could raise respondent burden and thus potentially increase panel attrition and nonresponse bias. To test this possibility, we use an experimental split questionnaire design administering two tests to one half of the respondents but only one test to the other half. Analyzing the effects of these different experimental settings on response behavior shows that even though those assigned to the longer test variant had no higher probability of refusing to participate in the survey, they were significantly more likely to apply alternative strategies for not taking the tests.

The Effect of Burdensome Survey Questions on Data Quality in an Omnibus Survey

“…Respondents can experience burden from knowledge questions for two reasons; respondents are aware that there is a correct response that they are evaluated on, which can contribute stress, and if respondents are aware of any poor performance on these questions, they may feel frustrated (Kleinert et al 2019). However, while there has been extensive research on how to design knowledge questions (e.g., Mondak 1999;Mondak 2001;Sturgis et al 2008), less attention has been paid to the degree of burden experienced by respondents when encountering these questions.…”

The effects of a special sequential mixed-mode design, and reminders, on panellists’ participation in a probability-based panel study

“…The timing of survey participation is thus associated with interviewees’ educational level (Green 1996 ): that is, a higher level of intelligence and achievement is associated with their earlier response after survey launch. Previous research provides evidence that lower educated persons are less likely to participate in surveys “because they have lower appreciation for this type of research” (Kleinert et al 2019 : 22; Revilla 2012 ). Therefore, in line with Olson et al ( 2012 ), it is assumed that the panellists’ mode preference, indicated indirectly by their cognitive abilities and social resources, predicts participation in web and phone modes, and the selection of a mode when given the option of two modes.…”

“…a sequential mixed-mode design with push-to-web procedure—takes into account invitees’ willingness and decision to participate in the survey (Becker et al 2019 ; Dillman et al 2014 ; Singer 2011 ; Groves and Couper 1998 ). On the one hand, the decision on survey participation is facilitated and the cognitive burden of choosing among different modes is avoided (Kleinert et al 2019 ; de Leeuw 2018 ; Lenzner et al 2009 ). On the other hand, due to widespread internet use in the target population and the seemingly lower burden and effort for respondents, this is an elaborate and efficient survey mode in multiple-panel studies (Shih and Fan 2008 ; Couper and Miller 2009 ; Fan and Yan 2010 ; Tourangeau et al 2013 ; Göritz 2014 ).…”

Analyzing Nonresponse in Longitudinal Surveys Using Bayesian Additive Regression Trees: A Nonparametric Event History Analysis

“…Panel studies require repeated participation of sampling units across long periods of time. However, many respondents are reluctant to invest the sustained effort required for this task and refuse follow-up invitations to surveys (e.g., Kleinert et al, 2019; Williams & Brick, 2017). Unit nonresponse resulting from a refusal to participate in a study is commonly referred to as dropout, break off, or attrition (Brüderl & Trappmann, 2017; Peytchev, 2009) and represents an increasing problem in social science research.…”