Making the Most of Imperfect Data: A Critical Evaluation of Standard Information Collected in Farm Household Surveys

Abstract

SUMMARYHousehold surveys are one of the most commonly used tools for generating insight into rural communities. Despite their prevalence, few studies comprehensively evaluate the quality of data derived from farm household surveys. We critically evaluated a series of standard reported values and indicators that are captured in multiple farm household surveys, and then quantified their credibility, consistency and, thus, their reliability. Surprisingly, even variables which might be considered ‘easy to estimate’ had instances of non-credible observations. In addition, measurements of maize yields and land owned were found to be less reliable than other stationary variables. This lack of reliability has implications for monitoring food security status, poverty status and the land productivity of households. Despite this rather bleak picture, our analysis also shows that if the same farm households are followed over time, the sample sizes needed to detect substantial changes are in the order of hundreds of surveys, and not in the thousands. Our research highlights the value of targeted and systematised household surveys and the importance of ongoing efforts to improve data quality. Improvements must be based on the foundations of robust survey design, transparency of experimental design and effective training. The quality and usability of such data can be further enhanced by improving coordination between agencies, incorporating mixed modes of data collection and continuing systematic validation programmes.

Reducing agro-environmental trade-offs through sustainable livestock intensification across smallholder systems in Northern Tanzania

“…Moreover, quality of outputs entirely depends on the quality of household survey data, which has often been questioned. Key data including land and plot sizes and yields are often over-or under-estimated by farmers themselves (Carletto, Zezza, & Banerjee, 2013;Fraval et al, 2019). Modelling few farming systems, and participatory validation and triangulation of input data using mixedmethods including actual on-farm measurements, such as performed in this study, enables more indepth understanding of complexities, underlying dynamics and relationships between farming systems components.…”

The Rural Household Multiple Indicator Survey, data from 13,310 farm households in 21 countries

“…The tool has been systematically designed to enable the quantification of interactions between different components and outcomes of agricultural systems, including productivity, and human welfare at the farm and household level, and it has been widely adopted by research organisations and development partners 8 . Such a streamlined, modular approach has resulted in a strong reduction in costs 9 compared to traditional households surveys in the field (which in other approaches typically take 2-3 hours per household 10 ) and of the subsequent data analysis and reporting 11 .…”

Experiences and Drivers of Food Insecurity in Guatemala's Dry Corridor: Insights From the Integration of Ethnographic and Household Survey Data

“…Within a capabilities framing of food insecurity, survey instruments that include multiple indicators (e.g., RHoMIS) are useful because standardization offer a means to replicable, statistical analysis, while multiple indicators work toward more holistic measurement of household capabilities and dimensions of food security outcomes. Standardization enables the relative comparison of socio-economic status and food insecurity outcomes spatially (e.g., between populations or regions) and temporally (e.g., before and after an intervention), and for these associations to be tested statistically (Fraval et al, 2018). The World Food Summit (1996) provided a definition of food security as being a condition in which "all people, at all times, have physical and economic access to sufficient, safe and nutritious food to meet their dietary needs and food preferences for an active and healthy life."…”

“…However, simply extending surveys to be all encompassing is unlikely to be a feasible response to the complexity and context dependency of the lived experience of food insecurity documented in this region, due to the pressure that would put on data quality, for example by increasing time-cost, participant fatigue and recall accuracy (Kilic and Sohnesen, 2015). Before extending surveys, it is important to address the current limitations to produce insight from surveys due to issues of data quality and biases, as detailed in Fraval et al (2018). We also note that some experiences and topics do not fit the standardized structure of a survey.…”

“…Under time and resource restrictions the application of ethnographic methods to a stratified subset of surveyed households manifests as a trade-off to the sample size of the larger survey effort. However, a critical evaluation of survey data was able to capture significant differences and trends in survey indicators that are representative of the wider population using sample sizes of hundreds, while many survey efforts typically reach into the thousands suggesting there is room for the inclusion of more mixed methods in standardized assessment and monitoring protocol (Fraval et al, 2018).…”