Treatment Effect Heterogeneity

Abstract

This paper considers recent methodological developments in the treatment effects literature, describes their value for applied evaluation work, and suggests next steps. It pays particular attention to documenting the presence of treatment effect heterogeneity, to the quest to attach treatment effect heterogeneity to particular subgroups and other moderators, and to the recent application of machine learning methods in this domain.

Associations among educators’ beliefs, intervention fidelity, and student outcomes in school-wide positive behavior interventions, and supports: A school-level moderated mediation analysis.

“…This pragmatic study was not initially designed to power moderated mediation analysis. However, our preliminary findings supported the utility of equity-informed analytic practices to (a) advance our precise understanding of for whom and how an implementation strategy works (Smith, 2022) and (b) inform the development, test, and refinement of equitable implementation strategies for diverse school populations and settings (Kazdin, 2007).…”

Causal complexity in human research: On the shared challenges of behavior genetics, medical genetics, and environmentally oriented social science

“…This is perhaps most obvious in the case of therapeutic and educational interventions delivered one-on-one. In an RCT of cognitive behavioral therapy, for instance, the “underlying treatment … may in a real sense differ for every single unit,” because the content of every session is tailored specifically to the individual (Smith, 2022, p. 656). But even interventions that are not psychotherapeutic might be, in practice, implemented in a highly idiosyncratic way (see, e.g., an ethnography of welfare case workers by Watkins-Hayes, 2009).…”

“…Thus, even as philosophers and plant geneticists extol the homogeneity of environmental interventions in the social and behavioral sciences, interventionists themselves paint quite a different picture: “We must expect, study and capitalize on the heterogeneity that characterizes most effects in science” (Bryan, Tipton, & Yeager, 2021, p. 986). In fact, environmental interventionists sound remarkably like behavioral geneticists: “The researcher faces a tough trade-off between interpretability and statistical power or, put differently, between learning about the effects of the underlying heterogeneous treatments and the sample size available for studying each treatment” (Smith, 2022, p. 656). These quotes illustrate that the problem of causal stimulus heterogeneity, while definitely a formidable challenge to mechanistic understanding, is not a challenge that is unique to the study of genetic causes, but is rather a difficulty that besets most studies in the social and behavioral sciences.…”

Assessing the properties of patient-specific treatment effect estimates from causal forest algorithms under essential heterogeneity

“…Under the assumption of no omitted variable bias, regression-based estimators yield unbiased estimates of the average treatment effect for the subset patients who chose treatment or the average treatment effect on the treated (ATT) [ 43 , 48 – 50 , 54 , 57 , 60 , 68 , 69 ]. Consequently, if treatment choice in an empirical setting was influenced by unmeasured patient factors related to treatment effectiveness – essential heterogeneity – the parametric estimate of ATT for a reference class will overstate the true treatment effects for the untreated patients in the class [ 39 , 49 , 50 , 70 ]. Researchers using parametric estimators have learned not to generalize a single parametric treatment effect estimate to all patients in a population [ 38 , 43 , 47 – 51 , 53 , 55 , 56 , 58 , 59 , 61 , 67 , 70 , 71 ].…”

“…Consequently, if treatment choice in an empirical setting was influenced by unmeasured patient factors related to treatment effectiveness – essential heterogeneity – the parametric estimate of ATT for a reference class will overstate the true treatment effects for the untreated patients in the class [ 39 , 49 , 50 , 70 ]. Researchers using parametric estimators have learned not to generalize a single parametric treatment effect estimate to all patients in a population [ 38 , 43 , 47 – 51 , 53 , 55 , 56 , 58 , 59 , 61 , 67 , 70 , 71 ].…”