Estimating complier average causal effects for clustered RCTs when the treatment affects the service population

Abstract

Randomized controlled trials (RCTs) sometimes test interventions that aim to improve existing services targeted to a subset of individuals identified after randomization. Accordingly, the treatment could affect the composition of service recipients and the offered services. With such bias, intention-to-treat estimates using data on service recipients and nonrecipients may be difficult to interpret. This article develops causal estimands and inverse probability weighting (IPW) estimators for complier populations in these settings, using a generalized estimating equation approach that adjusts the standard errors for estimation error in the IPW weights. While our focus is on more general clustered RCTs, the methods also apply (reduce) to nonclustered RCTs. Simulations show that the estimators achieve nominal confidence interval coverage under the assumed identification conditions. An empirical application demonstrates the methods using data from a large-scale RCT testing the effects of early childhood services on children’s cognitive development scores. An R program for estimation is available for download.

Estimating average treatment effects for clustered RCTs with recruitment bias

“…The appendix also displays the overlap in the estimated propensity score distributions of the treatment and control group samples using the Shaikh et al approach 43 F I G U R E D.1 Propensity score frequency distributions using a variant of the Shaikh et al 43 approach adapted to IPW models by Schochet. 24…”

“…Relatedly, Schochet 27 develops a principal stratification estimator for a SACE‐related estimand that adjusts for mobility bias in longitudinal clustered RCTs. Further, Ding and Lu 28 and Schochet 24 develop IPW methods for RCTs for mediational analyses to estimate causal effects of intermediate variables (such as the receipt of a specific class of services) on longer‐term outcomes, where the mediator values could be affected by the intervention. The IPW estimators for these studies exploit the balanced ITT samples as well as the availability of baseline data for each mediator group (eg, for both service recipient and non‐recipients).…”

“…Standard specification tests for the logit models can be used to assess treatment‐control balance, such as examining standardized covariate differences 41 . The Shaikh et al test 42 adapted to IPW settings 24 is also useful for assessing the overlap between $$$ {\hat{f}}_{q=1}(z) $$$ and $$$ \hat{\omega}\frac{z}{\left(1-z\right)}{\hat{f}}_{q=0}(z) $$$ for $$$ 0<z<1 $$$, where $$$ {\hat{f}}_{q=1}(z) $$$ and $$$ {\hat{f}}_{q=0}(z) $$$ are empirical densities of the estimated propensity scores for the two samples, and $$$ \omega =\frac{\mathit{\Pr}\left(q=0|r=1\right)}{\mathit{\Pr}\left(q=1|r=1\right)} $$$ is the population size ratio that can be estimated using sample counts. Confidence in the results can also be increased if measures of association between actual and predicted $$$ {q}_i $$$ values are high (relative to random allocations).…”

“…We use robust generalized estimating equation (GEE) methods for SE estimation that incorporate both estimation error in the IPW weights and design effects due to the clustered RCT design. [20][21][22][23][24] We conduct simulations to assess confidence interval coverage of the IPW estimators. We also conduct an illustrative empirical analysis to demonstrate the methods using data from a large-scale clustered RCT, co-funded by two federal U.S. agencies, testing the effects of school-based interventions to promote behavioral health.…”

“…We discuss specification tests for the logit propensity score models used to generate the IPW weights, as well as a simple data collection strategy to improve the predictive accuracy of the logit models. We use robust generalized estimating equation (GEE) methods for SE estimation that incorporate both estimation error in the IPW weights and design effects due to the clustered RCT design 20‐24 . We conduct simulations to assess confidence interval coverage of the IPW estimators.…”