Deriving Priors for Bayesian Prediction of Daily Response Propensity in Responsive Survey Design: Historical Data Analysis Versus Literature Review

Abstract

Responsive survey design (RSD) aims to increase the efficiency of survey data collection via live monitoring of paradata and the introduction of protocol changes when survey errors and increased costs seem imminent. Daily predictions of response propensity for all active sampled cases are among the most important quantities for live monitoring of data collection outcomes, making sound predictions of these propensities essential for the success of RSD. Because it relies on real-time updates of prior beliefs about key design quantities, such as predicted response propensities, RSD stands to benefit from Bayesian approaches. However, empirical evidence of the merits of these approaches is lacking in the literature, and the derivation of informative prior distributions is required for these approaches to be effective. In this paper, we evaluate the ability of two approaches to deriving prior distributions for the coefficients defining daily response propensity models to improve predictions of daily response propensity in a real data collection employing RSD. The first approach involves analyses of historical data from the same survey, and the second approach involves literature review. We find that Bayesian methods based on these two approaches result in higher-quality predictions of response propensity than more standard approaches ignoring prior information. This is especially true during the early-to-middle periods of data collection, when survey managers using RSD often consider interventions.

Some Open Questions on Multiple-Source Extensions of Adaptive-Survey Design Concepts and Methods

“…In addition to data that enrich the frame, paradata (Couper 2000(Couper , 2017 are also vital for estimating response propensities, and thus to inform interventions during the data collection period (Groves and Heeringa 2006;West et al 2021). Predicted or imputed responses, or information similar to key survey items, may be compared to accumulating response data to evaluate the quality of responses, and thus to inform intervention decisions (Morris et al 2015).…”

Comparing the Ability of Regression Modeling and Bayesian Additive Regression Trees to Predict Costs in a Responsive Survey Design Context

“…The cost of the post-paid incentive is a function of the number of interviews. We use response propensity models (described elsewhere, see West et al 2019) as the basis of estimating these costs. Our primary focus in this study is on prediction of the hours that interviewers will expend once the second phase begins.…”

Predicting Days to Respondent Contact in Cross-Sectional Surveys Using a Bayesian Approach

“…More recent applications have incorporated both historical data and data from the inprogress data collection period in a Bayesian framework to improve predictions of response propensities (Schouten et al 2018;West et al 2021). Both studies found that Bayesian methods offered modest improvements in predictions of daily response propensities compared to predictions relying on either historical or current data alone, particularly when model misspecification was small.…”