Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames

Abstract

The increasing use of internet surveys for stated preference studies raises questions about the effect of the survey mode on welfare estimates. A number of studies have conducted convergent validity investigations that compare the use of the internet with other survey implementation modes such as mail, telephone and in-person. All, but one, of these comparison studies is confounded different sample frames for the different modes of survey implementation. In this study we investigate differences in internet and mail survey modes holding the sample frame constant. This is done in the context of a choice-modelling study of improvements in the River Murray in Australia. We also investigate sample frame holding the survey mode (mail) constant. We find that survey mode (internet vs. mail) influences welfare estimates, and the internet welfare estimates are about 78 % of the mail welfare estimates on average. There is not a significant effect of sample frame (internet panel vs. postal addresses) on welfare estimates for implementation of a mail survey.

Impact of Survey Administration Mode on the Results of a Health-Related Discrete Choice Experiment: Online and Paper Comparison

“…For instance, Windle and Rolfe [24] reported $70 per paper survey and $15 per online survey [24]. In contrast to previous studies in which lower (average ratio 0.76) [12] and similar WTP estimates were found [24,25] for online responders versus paper responders, we found some of the WTP estimates in our study to be slightly higher for online respondents than for paper respondents, whereas other WTP estimates were equal between the samples. This study was subject to limitations.…”

“…We are aware of one additional published study comparing paper with online DCE administration since this review. This study by Boyle et al [12] showed that Internetbased welfare estimates were 76% of the mail-based welfare estimates on average. The health-related study described in our article is a novel contribution to the already existing literature because we study mode effects using a large number of respondents drawn from the same sample frame (an Internet panel), who completed a nearly identical survey either online or on paper during the same period.…”

Contemporary Guidance for Stated Preference Studies

“…Concerns regarding the use of the internet-based methods for surveys revolve around issues of representation of the population given varying levels of computer literacy across age groups, varying degrees of connectivity to the internet across different sociodemographic or economic groups, and the extent to which respondents "take care" when answering the questionnaire. Lindhjem and Navrud (2011b) reviewed 17 SP studies in which comparisons between internet and other survey formats are made and found only minor differences in values (see also Boyle et al 2016;Sandorf, Aanesen, 32. Moreover, cell phone numbers cannot be as readily assigned to specific geographic areas, making it harder to match the respondent to their neighborhood's characteristics.…”

What can discrete choice experiments do for you?

“…Researchers must, however, be aware that using different modes (e.g. mail, Internet, interviews) to collect data can lead to variations in representativeness, convergent validity and data quality . The choice of survey mode will depend on the research question and population of interest and findings should be interpreted with mode effects in mind.…”