Causal inference methods to study nonrandomized, preexisting development interventions

Abstract

Empirical measurement of interventions to address significant global health and development problems is necessary to ensure that resources are applied appropriately. Such intervention programs are often deployed at the group or community level. The gold standard design to measure the effectiveness of communitylevel interventions is the community-randomized trial, but the conditions of these trials often make it difficult to assess their external validity and sustainability. The sheer number of community interventions, relative to randomized studies, speaks to a need for rigorous observational methods to measure their impact. In this article, we use the potential outcomes model for causal inference to motivate a matched cohort design to study the impact and sustainability of nonrandomized, preexisting interventions. We illustrate the method using a sanitation mobilization, water supply, and hygiene intervention in rural India. In a matched sample of 25 villages, we enrolled 1,284 children <5 y old and measured outcomes over 12 mo. Although we found a 33 percentage point difference in new toilet construction [95% confidence interval (CI) = 28%, 39%], we found no impacts on height-for-age Z scores (adjusted difference = 0.01, 95% CI = −0.15, 0.19) or diarrhea (adjusted longitudinal prevalence difference = 0.003, 95% CI = −0.001, 0.008) among children <5 y old. This study demonstrates that matched cohort designs can estimate impacts from nonrandomized, preexisting interventions that are used widely in development efforts. Interpreting the impacts as causal, however, requires stronger assumptions than prospective, randomized studies.impact evaluation | study design | propensity score matching | communityled total sanitation | open defecation

Use, microbiological effectiveness and health impact of a household water filter intervention in rural Rwanda—A matched cohort study

“…Because the approach can be used to evaluate previously-delivered interventions, it may reduce the risk present in prospective randomized controlled trials where the intervention being evaluated is not delivered or embraced as intended. Its purpose is to match on potential confounders at the design phase in order to achieve a valid comparison group, reducing the bias and strengthening causal inference without relying on post-hoc modelling assumptions (Arnold et al 2010). "…”

“…Village-level matching was performed using a combination of restriction, propensity score matching, and rapid assessment (Arnold et al, 2009(Arnold et al, , 2010. Intervention villages were first exact matched to non-bordering potential control villages within the same health centre catchment area (sub-district).…”

“…The matched cohort design seeks to minimise the risk of unmeasured confounders by matching on characteristics likely to impact outcomes of interest (Austin, 2011;Stuart, 2010). This design has been used in other studies of pre-existing interventions where randomization is not possible (Arnold et al, 2009(Arnold et al, , 2010Ercumen et al, 2015a).…”

H2S as an Indicator of Water Supply Vulnerability and Health Risk in Low-Resource Settings: A Prospective Cohort Study

“…We conducted this analysis as part of a 12-month longitudinal cohort study in rural Southern India, designed to measure the effect of a combined sanitation, water supply, and hygiene intervention program on child health. 25 …”

The Impact of Conditional Cash Transfer on Toilet Use in eThekwini, South Africa

“…Regardless of the inconsistencies, governments are judged based on the number of structures they provide [20], subsidies persist and are undeniably effective at increasing coverage [23]. Still, access does not necessarily imply use, and even if use begins high, it often declines rapidly over time, especially once the daily burden of cleaning and maintenance become evident [24][25][26][27]. After decades of counting toilets, advanced methods to by-pass unreliable stated use measurements or invasive observations are now only in development [28].…”