Errors in Survey Reporting and Imputation and Their Effects on Estimates of Food Stamp Program Participation

Abstract

Accurately measuring government benefit receipt in household surveys is necessary when studying disadvantaged populations and the programs that serve them. The Food Stamp Program is especially important given its size and recent growth. To validate survey reports, we use administrative data on participation in two states linked to the American Community Survey (ACS), the Current Population Survey (CPS), and the Survey of Income and Program Participation (SIPP). We find that 23 percent of true food stamp recipient households do not report receipt in the SIPP, 35 percent in the ACS, and fully 50 percent in the CPS. A substantial number of true non-recipients are also recorded as recipients, especially in the SIPP. We examine reasons for these errors including imputation, an important source of error. Both false negative and false positive reports vary with household characteristics, implying complicated biases in multivariate analyses, such as regressions. We then directly examine biases in common survey-based estimates of program receipt by comparing them to estimates from our combined administrative and survey data. We find that the survey estimates understate participation among single parents, non-whites, and low-income households, and also lead to errors in multiple program receipt, and time and age patterns of receipt.

The Poverty Reduction of Social Security and Means-Tested Transfers

“…Our estimate for the poverty reduction of SNAP also exceeds that of Tiehen et al (2015) by 41%. This pattern is consistent with existing evidence showing that the CPS markedly understates SNAP receipt (see Meyer et al 2009;Meyer, Mittag, and Goerge 2018). Our estimate of the poverty reduction of housing assistance also exceeds those of Short (2012) and Fox (2017) by 63% and 32%, respectively.…”

SNAP timing and food insecurity

“…We show that if we assume MTR and MTS we can get informative bounds on the potential decreases in very low food security due to SNAP participation. Because program participation is mis-measured in survey data -mostly, under-reported [23][24][25][26]-we also estimate how these bounds vary with different amounts of assumed under-reporting in SNAP participation.…”

Combining Administrative and Survey Data to Improve Income Measurement

“…Andridge and Little (2010) and Little and Rubin (2002). Recent studies provide evidence that imputation can induce substantial error in survey data (Meyer, Mittag, and Goerge, 2018;Celhay, Meyer, and Mittag 2018a) and in estimates derived from them (Lillard, Smith, and Welch, 1986;Hirsch and Schumacher, 2004;Bollinger and Hirsch, 2006). Celhay, Meyer and Mittag (2018a) use linked data to examine how imputations of transfer receipt differ from the accurate administrative variable.…”

“…Even worse, the studies mentioned above show that response errors are not independent of other respondent characteristics or the true value of the variable, so that they likely bias both causal and descriptive estimates obtained from survey data (e.g. Bollinger and David 1997;Bound, Brown, and Mathiowetz 2001;Meyer, Mittag and Goerge, 2018;Mittag, 2017, 2019b). For example, Meyer, Mittag and Goerge (2018) find that survey error leads to biased estimates of the determinants of program receipt.…”

“…Therefore, linking program records can provide us with an accurate measure of a survey concept. Past research on government transfer programs, such as Medicaid (Davern et al, 2008), the Supplemental Nutrition Assistance Program (SNAP, formerly The Food Stamp Program, see Bollinger and David, 1997, Celhay, Meyer and Mittag 2018a, Harris, 2014, Kirlin and Wiseman 2014, Marquis and Moore 1990, Meyer, Mittag and Goerge 2018, Taeuber et al 2004, cash welfare Mittag 2018a, Lynch et al 2007) and social security and pensions (Nicholas and Wiseman 2009, Gathright and Crabb 2014, Bee and Mitchell 2017a, has used data linkage to demonstrate substantial survey error that has important effects on survey estimates. Both policy makers and academics heavily rely on the data in which these studies document sizeable error.…”