Estimating Propensity Adjustments for Volunteer Web Surveys

Abstract

Panels of persons who volunteer to participate in Web surveys are used to make estimates for entire populations, including persons who have no access to the Internet. One method of adjusting a volunteer sample to attempt to make it representative of a larger population involves randomly selecting a reference sample from the larger population. The act of volunteering is treated as a quasi-random process where each person has some probability of volunteering. One option for computing weights for the volunteers is to combine the reference sample and Web volunteers and estimate probabilities of being a Web volunteer via propensity modeling. There are several options for using the estimated propensities to estimate population quantities. Careful analysis to justify these methods is lacking. The goals of this article are (a) to identify the assumptions and techniques of estimation that will lead to correct inference under the quasi-random approach, (b) to explore whether methods used in practice are biased, and (c) to illustrate the performance of some estimators that use estimated propensities. Two of our main findings are (a) that estimators of means based on estimates of propensity models that do not use the weights associated with the reference sample are biased even when the probability of volunteering is correctly modeled and (b) if the probability of volunteering is associated with analysis variables collected in the volunteer survey, propensity modeling does not correct bias.

#Step @by #Step: recommendations for the development of high quality online research

“…To meet the need for psychographic or attitudinal variables, some companies and institutes use Reference Surveys, which are also discussed in the literature as alternatives for bias reduction (Valliant & Dever, 2011).…”

Is the web a promising tool for data collection in developing countries? An analysis of the sample bias of 10 web and face-to-face surveys from Africa, Asia, and South America

“…As a consequence, it seems that, considering the absence of an adequate sampling International Journal of Social Research Methodology 3 frame and the applied self-selection recruitment methods, the basic methodological condition for generalizing for the whole population can hardly be met for a volunteer web survey. While the question of representativeness and methodological solutions for volunteer web surveys has been analyzed predominantly for one or a few developed countries (Bethlehem, 2009;Bethlehem & Biffignandi, 2012;Lee, 2006;Loosveldt & Sonck, 2008;Schonlau, van Soest, Kapteyn, & Couper, 2009;Valliant & Dever, 2011), the question remains whether similar patterns of sample bias can be detected for developing countries. An answer to this question is particularly urgent, as emerging web surveys in developing countries will more often be of a non-probability nature due to the lack of proper sampling frames, but when sampling frames improve, letter invitations to participate in a survey will more often be used.…”

The Use of a Nonprobability Internet Panel to Monitor Sexual and Reproductive Health in the General Population

“…Furthermore, using a propensity score (in complement of a calibration procedure or not) is not a guarantee to get smaller biases or variances and there are limited evidence that using nondemographic variables is of great help (AAPOR 2013;Lee 2006). We also used a very large set of sociodemographic variables for our calibration, some of them being clearly linked to the variables of interest (age, household composition, education, and employment) that may lead to substantial bias and variance reduction (Dever et al 2008;Valliant and Dever 2011).…”

“…These latter variables, generally opinions on various social subjects or Internet practice, may help correcting the selection bias of the online recruitment and of the online survey. However, a poststratification or a calibration procedure is still needed after or before this method-evidences are not very clear on the order (Loosveldt and Sonck 2008;Valliant and Dever 2011). The conclusion of the AAPOR report is that the results of both methods are mitigated, depending on the thematic of the surveys, the variables used for the weighting, the online survey characteristics, and so on.…”