Should We Offer Web, Paper, or Both? A Comparison of Single- and Mixed-Response Mode Treatments in a Mail Survey

Abstract

This article leverages a five-treatment response mode experiment (paper-only, web-only, sequential web-mail, choice, and choice-plus [choice with a promised incentive for responding online]) that was conducted within a nationally representative survey. Because this survey’s sample was drawn from respondents to another nationally representative survey, we have rich frame data that includes multiple indicators of comfort using the internet for our sample members and we can compare their response behavior across two surveys. We find that the paper-only treatment yielded a lower response rate than most of the other treatments, but there were not significant differences between the response rates for the other treatments. Among our mixed-mode treatments, the sequential web-mail treatment had the highest percentage of response by web and the lowest cost per response. When focusing on the subgroups that we expected to be the least—and the most—comfortable with the internet, we found that the paper-only treatment generally performed worse than the others, even among subgroups expected not to be comfortable with the internet. We generally did not find significant differences in the effect of response mode treatment on the response rate or percentage of response by web between the subgroups who were the most and least comfortable with the internet. In terms of the consistency of response mode choice over time, our results suggest that some people respond consistently—but also that response mode preferences are weak enough that they can be influenced by the way in which the modes are offered. We ultimately recommend using a sequential web-mail design to minimize costs while still providing people who cannot or will not respond by web with another response mode option. We also find evidence that there may be a growing lack of interest in responding by paper; more research is needed in this area.