Nonparametric Tests for Treatment Effect Heterogeneity

Abstract

A large part of the recent literature on program evaluation has focused on estimation of the average effect of the treatment under assumptions of unconfoundedness or ignorability following the seminal work by Rubin (1974) and Rosenbaum and Rubin (1983). In many cases however, researchers are interested in the effects of programs beyond estimates of the overall average or the average for the subpopulation of treated individuals. It may be of substantive interest to investigate whether there is any subpopulation for which a program or treatment has a nonzero average effect, or whether there is heterogeneity in the effect of the treatment. The hypothesis that the average effect of the treatment is zero for all subpopulations is also important for researchers interested in assessing assumptions concerning the selection mechanism. In this paper we develop two nonparametric tests. The first test is for the null hypothesis that the treatment has a zero average effect for any subpopulation defined by covariates. The second test is for the null hypothesis that the average effect conditional on the covariates is identical for all subpopulations, in other words, that there is no heterogeneity in average treatment effects by covariates. Sacrificing some generality by focusing on these two specific null hypotheses we derive tests that are straightforward to implement.

Implications of heterogeneous impacts of protected areas on deforestation and poverty

“…First, if treatment effects are not constant in the population, then average treatment effects will mask potential variations in response to treatment from subgroups of the population. Therefore, in spite of the fact that the environmental and socioeconomic impacts of Bolivia's protected areas have been modest, on average, there may be large impacts within certain areas or subgroups [19]. Second, we echo the arguments of Ferraro & Hanauer [20] that evidence-based policy design can greatly benefit from understanding the conditions under which conservation efforts have been most and least successful in the past.…”

Integrated conservation and development: evaluating a community-based marine protected area project for equality of socioeconomic impacts

“…Ferraro & Hanauer [58] provide a number of recommendations for evaluations of heterogeneous impacts to help mitigate the likelihood of mislabelling spurious correlations, including: (i) selecting a small (less than 5) number of subgroups based on theory and policy relevance; (ii) proceeding with caution to assess heterogeneous impacts if an unconditional treatment effect is not found; (iii) conducting an omnibus test to test for heterogeneity across all subgroups (e.g. [60]), and (iv) maintaining a constant family-size error rate. Note that some of these recommendations (e.g.…”

Protecting Ecosystems and Alleviating Poverty with Parks and Reserves: ‘Win-Win’ or Tradeoffs?

“…We form an ex post control group, based on observable covariates, on which we conduct subgroup analyses with the ATT as the estimand of interest. Subgroup analyses are relatively rare in the program evaluation literature (Crump et al 2008), but can provide valuable insight even when average treatment effects are not significantly different from zero (Crump et al 2008;Imbens and Wooldridge 2009). Perhaps the most common method of subgroup analysis is the use of interaction terms in a regression framework.…”

“…However, even if this type of approach were preceded by matching (Ho et al 2007) or trimming (Imbens 2004;Imbens and Wooldridge 2009), the subgroup treatment effect estimate is more similar to the Average Treatment Effect (ATE) than the ATT. Crump et al (2008) suggest estimating separate regression functions (parametric or nonparametric) for treatment and control groups, and testing for differences in the coefficients on the subgroup variable. While this approach is more transparent, it too is an estimand that is more in-line with ATE than ATT.…”

“…The econometric and program evaluation literature tends to focus primarily on the estimation of mean treatment effects, paying little attention to the impacts of treatment on population subgroups (Manski 2005;Crump et al 2008). Yet, as noted by Manski (2005), average treatment effects may not provide sufficient information to a social planner whose goal is to maximize a specific social welfare function.…”