This article introduces and discusses a method for conducting an analysis of nonresponse for a longitudinal establishment survey using regression trees. The methodology consists of three parts: analysis during the frame refinement and enrollment phases, common in longitudinal surveys; analysis of the effect of time on response rates during data collection; and analysis of the potential for nonresponse bias. For all three analyses, regression tree models are used to identify establishment characteristics and subgroups of establishments that represent vulnerabilities during the data collection process. This information could be used to direct additional resources to collecting data from identified establishments in order to improve the response rate.

Increasing nonresponse rates in federal surveys and potentially biased survey estimates are a growing concern, especially with regard to establishment surveys. Unlike household surveys, not all establishments contribute equally to survey estimates. With regard to agricultural surveys, if an extremely large farm fails to complete a survey, the United States Department of Agriculture (USDA) could potentially underestimate average acres operated among other things. In order to identify likely nonrespondents prior to data collection, the USDA’s National Agricultural Statistics Service (NASS) began modeling nonresponse using Census of Agriculture data and prior Agricultural Resource Management Survey (ARMS) response history. Using an ensemble of classification trees, NASS has estimated nonresponse propensities for ARMS that can be used to predict nonresponse and are correlated with key ARMS estimates.

Respondent burden has important implications for survey outcomes, including response rates and attrition in panel surveys. Despite this, respondent burden remains an understudied topic in the field of survey methodology, with few researchers systematically measuring objective and subjective burden factors in surveys used to produce official statistics. This research was designed to assess the impact of proxy measures of respondent burden, drawing on both objective (survey length and frequency), and subjective (effort, saliency, and sensitivity) burden measures on response rates over time in the Current Population Survey (CPS). Exploratory Factor Analysis confirmed the burden proxy measures were interrelated and formed five distinct factors. Regression tree models further indicated that both objective and subjective proxy burden factors were predictive of future CPS response rates. Additionally, respondent characteristics, including employment and marital status, interacted with these burden factors to further help predict response rates over time. We discuss the implications of these findings, including the importance of measuring both objective and subjective burden factors in production surveys. Our findings support a growing body of research suggesting that subjective burden and individual respondent characteristics should be incorporated into conceptual definitions of respondent burden and have implications for adaptive design.

No abstract

Measurement error is an important aspect of total survey error and is discussed at length in the Journal of Official Statistics, but there is very little discussion and/or application of psychometric tools such as Classical Test Theory (CTT) or Item Response Theory (IRT) being used to evaluate measurement error of latent traits (traits that are not directly observed, like respondent burden). CTT and IRT are specifically designed to evaluate the validity and reliability of measures dealing with latent traits, and are commonly used to assess the validity and reliability of psychological and educational measures; however this does not appear to be a common practice in the construction of the survey questionnaires discussed in the pages of JOS. In fact, while fifty-five articles come up in JOS searching "measurement," only twelve come up searching "reliability," nine searching "validity," six searching "latent," four searching "item response theory," and zero articles come up when using the search terms "Classical Test Theory" or "psychometric." While psychometric assessment is commonly taught in education and psychological research methods programs, it is not generally understood outside of those fields. The authors of the book Statistical Analysis of Questionnaires: A Unified Approach Based on R and Stata, Bartoulucci, Bacci, and Gnaldi provide a rich and easy to follow overview of psychometric theory that can easily be understood and applied by survey methodologists outside of the fields of education and psychology; they discuss the theoretical framework behind these types of models as well as provide a practical guide for assessing questionnaire latent constructs using psychometric evaluation methods.Many of the surveys used outside of education and psychology also contain latent traits, traits that we cannot directly observe, but can indirectly measure using a series of related questions (i.e., respondent burden). CTT and IRT can be used to compare and contrast the relationship between items and/or factors to assess and compare the reliability of items or latent construct measures as a whole. CTT assumes that the standard error is the same for all scores in a given population, where IRT assumes the standard errors vary. The authors discuss psychometrics from both a CTT and an IRT perspective. The book is designed specifically for graduate psychometric and statistics courses with an emphasis on measurement via questionnaires, but can be used by any survey methodologist and or practitioner interested in evaluating the reliability of latent construct measurement in their survey. The book is not only rich in theory and provides a thorough background, but it also q Statistics Sweden