Selection Error in Single- and Mixed Mode Surveys of the Dutch General Population

Abstract

Summary This study compares the extent of selection error (non‐response and coverage error) evoked by the four major contemporary modes of data collection (face to face, telephone, mail and Web) and three sequential mixed mode designs (telephone, mail and Web with face‐to‐face follow‐up) for the case of the Dutch Crime Victimization Survey. Sociodemographic characteristics and target variables from the survey serve as benchmark variables. A special two‐wave experimental design allows studying design differences in selection error on Crime Victimization Survey variables independently from differences in measurement error. Despite large differences in response rates, only small or no differences in selection error between the four single‐mode designs are found on both types of variable. We observe cases when the error is enlarged or mitigated in the mixed mode designs despite the fact that the designs yielded large response increases. Our results question the use of response rates to motivate the choice of mode and use of mixed mode surveys.

Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes

“…However, few studies have attempted to estimate the magnitude of selection errors before and after switching modes, while controlling for measurement differences. One recent study using a combination of data from population registers, a single mode benchmark, and a re-interview design (Klausch et al 2015a) observed an increase in response rates as a result of mixing modes sequentially (a face-to-face follow-up of web, mail and CATI surveys), but found no consistent reduction in selection errors present in estimates from the starting modes. Selection error was, however, reduced for some estimates from the mixed mode surveys as a result of bringing it closer in line with the selection error of the single mode (face-to-face) benchmark (ibid.…”

“…As a result, the available evidence as to the effects of mixing modes on different error sources, as well as on the TSE, remains somewhat inconclusive. In relation to selection errors, for example, several studies have analysed response rates and sample composition as proxies, and have confirmed that both can improve in mixed mode designs compared to single mode designs, depending on which modes are combined and how (e.g., Dillman et al 2014;Eva et al 2010;Fowler et al 2002;Greene et al 2008;Link and Mokdad 2006;Millar and Dillman 2011;Lynn 2013;Klausch et al 2015a). However, few studies have attempted to estimate the magnitude of selection errors before and after switching modes, while controlling for measurement differences.…”

“…Different approaches to the problem of how to disentangle confounded mode effects on selection and measurement errors have been proposed Tourangeau 2017;Vannieuwenhuyze and Loosveldt 2012). Each approach depends on the availability of auxiliary data, which may already be available to researchers -such as register data (Klausch et al 2015a), or data from the recruitment wave of a longitudinal survey ) -or (more likely) need to be collected separately . The latter could include a randomised mode experiment embedded in a mixed mode design (De Leeuw et al 2008), a single mode follow-up of a random sample of respondents (Klausch et al 2014;Schouten et al 2013), or a new or existing single mode survey (ideally) conducted alongside the mixed mode survey that can serve as a benchmark Vannieuwenhuyze and Loosveldt 2012).…”

“…For example, to assess measurement bias, external records can be used to assess the accuracy of respondents' self-reports (e.g., Olson 2006;Kreuter et al 2008;Sakshaug et al 2010;Tourangeau et al 2010). To assess nonresponse bias, these auxiliary data are needed for both respondents and nonrespondents (e.g., Klausch et al 2015a;Kreuter et al 2010;Kappelhof 2013). As such data are rarely available to researchers, the potential utility of the MSE as a metric for evaluating the effects of different survey design features or for comparing whole survey systems (Biemer 1988) has not been fully exploited.…”

“…Notably, the choice of mode can affect (1) coverage error, by determining whether a population member has a chance of being selected to participate in a survey (e.g., if the sample design depends on a list of incomplete information needed to implement the survey in a particular mode (Carley-Baxter et al 2010); (2) nonresponse error, because selected sample members may not have the possibility to participate in the chosen survey mode, or may be more or less willing to participate depending on the mode offered (Klausch et al 2015a); (3) measurement error, because mode characteristics can influence how respondents come up with their answers to survey questions and the answers they give (Dillman et al 2014;De Leeuw 2005); and (4) processing error, because, for example, noncomputerised methods of data entry and coding are more vulnerable to human error, or because interviewers may be less accurate in recording the responses given by respondents than the latter would be themselves ). Furthermore, because data collection modes vary in terms of their associated fixed and variable costs (interviewer-administered modes being most expensive), mode choice partly determines the amount of (5) sampling error in statistics, because under a fixed budget constraint, a survey designer could afford to survey different sized samples using different modes (Vannieuwenhuyze 2014).…”