Empirical Performance of Covariates in Education Observational Studies

Abstract

This paper summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observational studies where the outcome is a standardized reading or math test. They are: pretest measures, local geographic matching, and rich covariate sets with a strong theory of treatment selection. Overall, the review demonstrates that although the pretest often reduces bias in observational studies, it does not always eliminate it. Its performance depends on the pretest's correlation with treatment selection and the outcome, and whether pre-intervention trends are present. We also find that although local comparisons are prioritized for matching, its performance depends on whether comparable no-treatment cases are available. Otherwise, local comparisons may produce badly biased results. In cases where researchers have a strong theory of selection and rich covariate sets, observational methods perform well, but additional replication studies are needed. Finally, observational methods that rely on demographic covariates without a theory of selection rarely produce unbiased treatment effects. The paper Downloaded by [University of Nebraska, Lincoln] at 05:40 13 June 2016 ACCEPTED MANUSCRIPT ACCEPTED MANUSCRIPT 2 concludes by offering education researchers empirically-based guidance on covariate selection in observational studies.

The Design of Clustered Observational Studies in Education

“…We discuss sensitivity analysis in Section 3.4. Although this assumption may seem implausible, in many examples, treatment assignment depends on observed data only (Dehejia & Wahba, 1999;Fralick et al, 2018;Hernán & Robins, 2016;Keele et al, 2020;Wong et al, 2017).…”

The Effects of Tulsa's Pre‐K Program on Middle School Student Performance

“…Fourth, we conduct a number of sensitivity tests, subjecting our conclusions to an extensive battery of validity checks. These techniques, when used in tandem, have considerable potential to reduce selection bias and enhance validity (Wong, Valentine, & Miller‐Baine, ).…”

Within study comparisons and risk of bias in international development: Systematic review and critical appraisal

“…MDRC Steiner and Assessing correspondence between experimental and nonexperimental results in within-study-comparisons EdPolicyWorks Working Paper Wong and Steiner (2016) Designs of empirical evaluations of nonexperimental methods in field settings EdPolicyWorks Working Paper Jaciw (2016) Assessing the accuracy of generalized inferences from comparison group studies using a within-study comparison approach: The methodology Evaluation Review Wong et al (2017) Empirical performance of covariates in education observational studies Methodological Studies Chaplin et al (2018) The internal and external validity of the regression discontinuity design: A meta-analysis of 15 within-study-comparisons Policy Analysis and Management methodological designs, such as RDD (Chaplin et al, 2018; and propensity score matching (PSM; Shadish, 2013), or map internal replication designs . Only one known review is dedicated to evidence from social and economic development programmes in L&MICs (Hansen et al, 2013).…”