An experimental evaluation of a stopping rule aimed at maximizing cost-quality trade-offs in surveys

Abstract

Surveys face difficult choices in managing cost-error trade-offs. Stopping rules for surveys have been proposed as a method for managing these trade-offs. A stopping rule will limit effort on a select subset of cases to reduce costs with minimal harm to quality. Previously proposed stopping rules have focused on quality with an implicit assumption that all cases have the same cost. This assumption is unlikely to be true, particularly when some cases will require more effort and, therefore, more costs than others. We propose a new rule that looks at both predicted costs and quality. This rule is tested experimentally against another rule that focuses on stopping cases that are expected to be difficult to recruit. The experiment was conducted on the 2020 data collection of the Health and Retirement Study (HRS). We test both Bayesian and non-Bayesian (maximum-likelihood or ML) versions of the rule. The Bayesian version of the prediction models uses historical data to establish prior information. The Bayesian version led to higher-quality data for roughly the same cost, while the ML version led to small reductions in quality with larger reductions in cost compared to the control rule.

Some Open Questions on Multiple-Source Extensions of Adaptive-Survey Design Concepts and Methods

“…Optimization provides a mathematical way to evaluate tradeoffs between estimated cost savings that come from reduced effort with expected increases in RMSE that come from lower response rates due to that reduced effort. Wagner et al (2023) also defined an optimization rule for identifying cases for intervention and sought to minimize a function of data collection costs and mean squared error (MSE) of several key survey variables. Optimization-based rules are complex to implement as they often require predictive models for survey estimates, response propensity, and data collection costs under different sets of data collection strategies.…”

“…Adaptive and responsive survey designs tailor data collection features to impact measures of data quality, such as nonresponse error or measurement error, or measures of cost (Schouten et al 2017). The tailoring decisions are anchored in specific pre-defined survey goals, such as increasing response rate or balance in the respondent population (Coffey et al 2020;Wagner et al 2012); reducing the variance of key survey estimates or the variation in weighting adjustments (Beaumont et al 2014;Paiva and Reiter 2017); or controlling specified data collection costs (Coffey and Elliott 2023;Peytchev 2014;Wagner et al 2023). As a result, adaptive and responsive designs typically increase effort (i.e., resources) to certain cases; and decrease effort in others.…”

“…Deterministic procedures, on the other hand, lead to a specific action, such as introducing a particular data collection feature, such as an incentive or a mailed questionnaire (Coffey et al 2015;Jackson 2020;Lavrakas et al 2018); introducing or withholding a mode of data collection in multi-mode surveys (Coffey et al 2020); or putting cases on hold or removing them from interviewer workloads to reduce effort applied to specific case (Coffey and Elliott 2023;Dahlhamer 2017;Tolliver et al 2017;Wagner et al 2023). The 2017 SIPP prioritization made use of deterministic business rules, in addition to its dynamic prioritization rules.…”