Design-Based Covariate Adjustments in Paired Experiments

Wu, Edward Ming-Yang; Gagnon-Bartsch, Johann A.

doi:10.3102/1076998620941469

Cited by 3 publications

(1 citation statement)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other randomization designs would call for modifications to the algorithm [e.g. Wu and Gagnon-Bartsch, 2021].…”

Section: Sample Splittingmentioning

confidence: 99%

Precise Unbiased Estimation in Randomized Experiments using Auxiliary Observational Data

Gagnon-Bartsch¹,

Sales²,

Wu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Randomized controlled trials (RCTs) are increasingly prevalent in education research, and are often regarded as a gold standard of causal inference. Two main virtues of randomized experiments are that they (1) do not suffer from confounding, thereby allowing for an unbiased estimate of an intervention's causal impact, and (2) allow for design-based inference, meaning that the physical act of randomization largely justifies the statistical assumptions made. However, RCT sample sizes are often small, leading to low precision; in many cases RCT estimates may be too imprecise to guide policy or inform science. Observational studies, by contrast, have strengths and weaknesses complementary to those of RCTs. Observational studies typically offer much larger sample sizes, but may suffer confounding. In many contexts, experimental and observational data exist side by side, allowing the possibility of integrating "big observational data" with "small but high-quality experimental data" to get the best of both. Such approaches hold particular promise in the field of education, where RCT sample sizes are often small due to cost constraints, but automatic collection of observational data, such as in computerized educational technology applications, or in state longitudinal data systems (SLDS) with administrative data on hundreds of thousand of students, has made rich, high-dimensional observational data widely available. We outline an approach that allows one to employ machine learning algorithms to learn from the observational data, and use the resulting models to improve precision in randomized experiments. Importantly, there is no requirement that the machine learning models * These authors contributed equally.

show abstract

“…Other randomization designs would call for modifications to the algorithm [e.g. Wu and Gagnon-Bartsch, 2021].…”

Section: Sample Splittingmentioning

confidence: 99%

Precise Unbiased Estimation in Randomized Experiments using Auxiliary Observational Data

Gagnon-Bartsch¹,

Sales²,

Wu³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Covariate adjustment in experiments with matched pairs

Bai,

Jiang,

Romano

et al. 2024

Journal of Econometrics

View full text Add to dashboard Cite

Precise unbiased estimation in randomized experiments using auxiliary observational data

Gagnon-Bartsch,

Sales,

et al. 2023

Journal of Causal Inference

View full text Add to dashboard Cite

Randomized controlled trials (RCTs) admit unconfounded design-based inference – randomization largely justifies the assumptions underlying statistical effect estimates – but often have limited sample sizes. However, researchers may have access to big observational data on covariates and outcomes from RCT nonparticipants. For example, data from A/B tests conducted within an educational technology platform exist alongside historical observational data drawn from student logs. We outline a design-based approach to using such observational data for variance reduction in RCTs. First, we use the observational data to train a machine learning algorithm predicting potential outcomes using covariates and then use that algorithm to generate predictions for RCT participants. Then, we use those predictions, perhaps alongside other covariates, to adjust causal effect estimates with a flexible, design-based covariate-adjustment routine. In this way, there is no danger of biases from the observational data leaking into the experimental estimates, which are guaranteed to be exactly unbiased regardless of whether the machine learning models are “correct” in any sense or whether the observational samples closely resemble RCT samples. We demonstrate the method in analyzing 33 randomized A/B tests and show that it decreases standard errors relative to other estimators, sometimes substantially.

show abstract

Design-Based Covariate Adjustments in Paired Experiments

Cited by 3 publications

References 29 publications

Precise Unbiased Estimation in Randomized Experiments using Auxiliary Observational Data

Precise Unbiased Estimation in Randomized Experiments using Auxiliary Observational Data

Covariate adjustment in experiments with matched pairs

Precise unbiased estimation in randomized experiments using auxiliary observational data

Contact Info

Product

Resources

About