Relationship Between Past Survey Burden and Response Probability to a New Survey in a Probability-Based Online Panel

Abstract

We conducted an idiographic analysis to examine the effect of survey burden, measured by the length of the most recent questionnaire, or number of survey invitations (survey frequency) in a one-year period preceding a new survey, on the response probability to a new survey in a probability-based Internet panel. The individual response process was modeled by a latent Markov chain with questionnaire length and survey frequency as explanatory variables. The individual estimates were obtained using a Monte Carlo based method and then pooled to derive estimates of the overall relationships and to identify specific subgroups whose responses were more likely to be impacted by questionnaire length or survey frequency. The results show an overall positive relationship between questionnaire length and response probability, and no significant relationship between survey frequency and response probability. Further analysis showed that longer questionnaires were more likely to be associated with decreased response rates among racial/ethnic minorities and introverted participants. Frequent surveys were more likely to be associated with decreased response rates among participants with a large household. We discuss the implications for panel management and advocate targeted interventions for the small subgroups whose response probability may be negatively impacted by longer questionnaires or frequent surveys.

Response Burden – Review and Conceptual Framework

“…A review of empirical literature shows that researchers have used four different approaches to measure response burden throughout four decades of research (see Table 1 in Yan et al (2020) for a summary of measurements of and uses of burden in empirical research). The first approach follows the meaning of the term "perceived burden" by directly asking respondents how burdensome the survey was (Bottone et al 2018;Galesic 2006;Hedlin et al 2008;Yan et al 2020). This single measurement of burden is found to be related to respondents' decision to break off a web survey (Galesic 2006), difficulty of contacting and recruiting respondents to take part in a survey (Yan et al 2020), the amount of item missingness (Yan et al 2020), and the level of respondents' concerns with a survey request (Yan et al 2020).…”

“…As a matter of fact, burden is used as a blanket term throughout the literature referring to both actual or objective burden and subjective or perceived burden. Researchers in the last four decades have used burden (e.g., Groves et al 1999), respondent burden (e.g., Bradburn 1977), response burden (e.g., Filion 1981), perceived burden (e.g., Hedlin et al 2008) and reporting burden (e.g., Eurostat 2011) in their work.…”

The Effects of Response Burden – Collecting Life History Data in a Self-Administered Mixed-Device Survey

“…Three of these elements can be seen as properties of a survey, while the amount of stress on the respondent is a product of the interaction between the survey contents and characteristics of the respondent. While more recent research has often relied on a respondent-centred approach in defining response burden (Hedlin et al 2008), we focus on properties of a survey, varying them experimentally.…”

Exploring Burden Perceptions of Household Survey Respondents in the American Community Survey

“…For household surveys, Sharp and Frankel (1983) found no evidence of a follow-up request impacting attitudes or behavior in a two-interview study and Read (2019) found that the burden of continued participation in a receipt scanning mobile app study was small. While there has also been discussion about the potential for an individual survey's contact strategy to contribute to burden (e.g., Hedlin et al 2008;Giesen et al 2011;Hughes et al 2016;Fricker 2016), there has been little examination of this, other than a question asked on the CE (U.S. Bureau of Labor Statistics 2012:…”