Design of two-stage experiments with an application to spillovers in tax compliance

Abstract

We set up a framework to conduct experiments for estimating spillover effects when units are grouped into mutually exclusive clusters. We improve upon existing methods by allowing for heteroskedasticity, intra-cluster correlation and cluster size heterogeneity, which are typically ignored when designing experiments. We show that ignoring these factors can severely overestimate power and underestimate minimum detectable effects. We derive formulas for optimal group-level assignment probabilities and the power function used to calculate power, sample size, and minimum detectable effects. We apply our methods to the design of a largescale randomized communication campaign in a municipality of Argentina to estimate total and neighborhood spillover effects on property tax compliance. Besides the increase in tax compliance of individuals directly targeted with our mailing, we find evidence of spillover effects on untreated individuals in street blocks where a high proportion of taxpayers were notified.

Inference for Two-stage Experiments under Covariate-Adaptive Randomization

“…Finally, we compare our inference method against various widely used regression based methods in a simulation study and an empirical application. One of the regression specification is equivalent to what has been proposed and studied in recent methodological papers on two-stage experiments by Basse and Feller (2018) and Cruces et al (2022). In our simulation study, we verify that our inference results are asymptotically exact, and find that statistical inference based on ordinary least squares regression could either be too conservative or invalid.…”

“…As for the size-weighted estimators, θP 2 and θS 2 are closely related but not numerically identical to the individual-weighted estimator proposed by Basse and Feller (2018) as they took a different perspective to incorporate weights into the estimators. Also, in Cruces et al (2022) and Vazquez-Bare (2022), they study the estimators obtained through a widely used saturated regression in multi-treatment experiments, which are identical to θP 2 and θS 2 when outcomes of all units from each cluster are observed or the number of observed units is proportional to the cluster size. Formally, when M g /N g = c for 0 < c < 1, θP 2 and θS 2 may be simultaneously obtained as the estimators of the coefficient on Z i,g and H g (1 − Z i,g ) through a regression like the following:…”

“…which is equivalent to the identifiable parts of the variance derived in Basse and Feller (2018) under finite population framework. Additionally, the asymptotic variance from Cruces et al (2022), which is derived for more general designs without using covariates, is closely related to (13) that can be viewed as an alternative expression under our specific settings.…”

“…Here, by a two-stage randomized experiment, we mean one in which clusters (e.g., households, schools, or graph partitions) are first randomly assigned to different levels of treated fraction and then units within each treated clusters are randomly assigned to treatment or control according to its selected treated fraction; by covariate-adaptive randomization, we mean randomization schemes that first stratify according to baseline covariates and then assign treatment status so as to achieve "balance" within each stratum. Two-stage randomized experiments are frequently used in social science (see for example Duflo and Saez (2003); Haushofer and Shapiro (2016); McKenzie and Puerto (2021)), and discussed by statisticians (see for example Hudgens and Halloran (2008); Basse and Feller (2018); Cruces et al (2022); Imai et al (2021)), as a general approach to causal inference with interference; that is, when one individual's treatment status affects outcomes of other individuals. Moreover, practitioners frequently use covariate information to design more efficient two-stage experiments (see for example Duflo and Saez, 2003;Beuermann et al, 2015;Ichino and Schündeln, 2012;Aramburu et al, 2019;Hidrobo et al, 2016;Kinnan et al, 2020;Malani et al, 2021;Muralidharan and Sundararaman, 2015;Banerjee et al, 2021;Rogers and Feller, 2018).…”

“…The analysis of data from two-stage randomized experiments and data from experiments with covariateadaptive randomization has received considerable attention, but most work has focused on only one of these two features at a time. Previous work on the analysis of two-stage randomized experiments includes Hudgens and Halloran (2008), Liu and Hudgens (2014), Rigdon and Hudgens (2015), Basse and Feller (2018), Basse et al (2019), Cruces et al (2022), Imai et al (2021) and Jiang et al (2022). Recent work on the analysis of covariate-adaptive experiments includes Bugni et al (2018), Bai et al (2021), Bai et al (2021), Bai et al (2022b) and Bai et al (2022a).…”