The Effect of Survey Mode on Data Quality: Disentangling Nonresponse and Measurement Error Bias

Abstract

More and more surveys are conducted online. While web surveys are generally cheaper and tend to have lower measurement error in comparison to other survey modes, especially for sensitive questions, potential advantages might be offset by larger nonresponse bias. This article compares the data quality in a web survey administration to another common mode of survey administration, the telephone. The unique feature of this study is the availability of administrative records for all sampled individuals in combination with a random assignment of survey mode. This specific design allows us to investigate and compare potential bias in survey statistics due to 1) nonresponse error, 2) measurement error, and 3) combined bias of these two error sources and hence, an overall assessment of data quality for two common modes of survey administration, telephone and web. Our results show that overall mean estimates on the web are more biased compared to the telephone mode. Nonresponse and measurement bias tend to reinforce each other in both modes, with nonresponse bias being somewhat more pronounced in the web mode. While measurement error bias tends to be smaller in the web survey implementation, interestingly, our results also show that the web does not consistently outperform the telephone mode for sensitive questions.

Prior Choice for the Variance Parameter in the Multilevel Regression and Poststratification Approach for Highly Selective Data. A Monte Carlo Simulation Study.

“…Unrealistically high values may be detected by comparing the variance parameters of different variables or in comparison to previous surveys or comparable applications. Comparing the weighting variables' distribution in the sample with external benchmarks (e.g., from official statistics) can serve as a first evaluation of the sample's selectivity (see, for example, Felderer, Kirchner, and Kreuter (2019)) and inform the decision on which prior strategy to use. In addition, measures such as described in Little et al (2019) might be applied.…”

What’s the Benefit of a Video? The Effect of Nonmaterial Incentives on Response Rate and Bias in Web Surveys

“…Further independent variables used to estimate the expected differences in response propensity between treatments include the potential participant’s gender (male/female), the highest level of education (measured in four categories), and the invitee’s migration background operationalized by citizenship status (Austrian or non-Austrian). To account for the possible age effects found elsewhere (Felderer et al, 2019; Shapiro-Luft & Cappella, 2013), we also include age in years as a control variable. 2…”

Recruiting a Probability-Based Online Panel via Postal Mail: Experimental Evidence

“…where the SE of the survey estimate SEtruey¯true^svy is estimated using a sample variance estimation approach which takes into account the complexity of the sampling design (see Appendix B for more information). Generally, the estimation of the SE of the relative bias is straightforward under the assumption that the benchmark values are true population characteristics (i.e., have no sampling variance; see Felderer et al, 2019, for a mathematical derivation).…”