Optimizing Data Collection Interventions to Balance Cost and Quality in a Sequential Multimode Survey

Abstract

High-quality survey data collection is getting more expensive to conduct because of decreasing response rates and rising data collection costs. Responsive and adaptive designs have emerged as a framework for targeting and reallocating resources during the data collection period to improve survey data collection efficiency. Here, we report on the implementation and evaluation of a responsive design experiment in the National Survey of College Graduates that optimizes the cost-quality tradeoff by minimizing a function of data collection costs and the root mean squared error of a key survey measure, self-reported salary. We used a Bayesian framework to incorporate prior information and generate predictions of estimated response propensity, self-reported salary, and data collection costs for use in our optimization rule. At three points during the data collection process, we implement the optimization rule and identify cases for which reduced effort would have minimal effect on the mean squared error (RMSE) of mean self-reported salary while allowing us to reduce data collection costs. We find that this optimization process allowed us to reduce data collection costs by nearly 10 percent, without a statistically or practically significant increase in the RMSE of mean salary or a decrease in the unweighted response rate. This experiment demonstrates the potential for these types of designs to more effectively target data collection resources to reach survey quality goals.

Some Open Questions on Multiple-Source Extensions of Adaptive-Survey Design Concepts and Methods

“…Adaptive and responsive survey designs tailor data collection features to impact measures of data quality, such as nonresponse error or measurement error, or measures of cost (Schouten et al 2017). The tailoring decisions are anchored in specific pre-defined survey goals, such as increasing response rate or balance in the respondent population (Coffey et al 2020;Wagner et al 2012); reducing the variance of key survey estimates or the variation in weighting adjustments (Beaumont et al 2014;Paiva and Reiter 2017); or controlling specified data collection costs (Coffey and Elliott 2023;Peytchev 2014;Wagner et al 2023). As a result, adaptive and responsive designs typically increase effort (i.e., resources) to certain cases; and decrease effort in others.…”

“…Response rates and mean costs-per-case were consistent across the group managed using adaptive procedures and the control group. Recently, Coffey and Elliott (2023) demonstrated, through the use of an optimization rule, that data collection costs in the NSCG could be reduced by nearly 10% versus the control group, without significant decreases in unweighted response rates nor increases in root mean squared error (RMSE) of the mean of a single key survey estimate, self-reported salary.…”

“…The last category of decision rules relies on optimization, by either maximizing or minimizing some function of survey data collection parameters to identify the optimal set of cases for intervention. Coffey and Elliott (2023) applied cost-and effort-reduction interventions to the set of cases that minimized the product of root mean squared error (RMSE) of the mean of a key survey statistic (salary) and data collection costs. Optimization provides a mathematical way to evaluate tradeoffs between estimated cost savings that come from reduced effort with expected increases in RMSE that come from lower response rates due to that reduced effort.…”

“…Deterministic procedures, on the other hand, lead to a specific action, such as introducing a particular data collection feature, such as an incentive or a mailed questionnaire (Coffey et al 2015;Jackson 2020;Lavrakas et al 2018); introducing or withholding a mode of data collection in multi-mode surveys (Coffey et al 2020); or putting cases on hold or removing them from interviewer workloads to reduce effort applied to specific case (Coffey and Elliott 2023;Dahlhamer 2017;Tolliver et al 2017;Wagner et al 2023). The 2017 SIPP prioritization made use of deterministic business rules, in addition to its dynamic prioritization rules.…”

“…CAES also hosts adaptive design models for various surveys such as the National Survey of College Graduates (NSCG; Coffey et al 2020), the SIPP (Tolliver et al 2017), the NTPS (Zotti 2019), and the Post-Enumeration Survey (PES). In the NSCG, for example, CAES has enabled the near-real time ingestion and processing of survey paradata and lightly edited response data to enable the estimation of multiple Bayesian predictive models to inform optimization decision rules about how to allocate effort to cases (Coffey and Elliott 2023).…”