Randomization Tests to Assess Covariate Balance When Designing and Analyzing Matched Datasets

Abstract

Causal analyses for observational studies are often complicated by covariate imbalances among treatment groups, and matching methodologies alleviate this complication by finding subsets of treatment groups that exhibit covariate balance. It is widely agreed upon that covariate balance can serve as evidence that a matched dataset approximates a randomized experiment, but what kind of experiment does a matched dataset approximate? In this work, we develop a randomization test for the hypothesis that a matched dataset approximates a particular experimental design, such as complete randomization, block randomization, or rerandomization. Our test can incorporate any experimental design, and it allows for a graphical display that puts several designs on the same univariate scale, thereby allowing researchers to pinpoint which design-if any-is most appropriate for a matched dataset. After researchers determine a plausible design, we recommend a randomizationbased approach for analyzing the matched data, which can incorporate any design and treatment effect estimator. Through simulation, we find that our test can frequently detect violations of randomized assignment that harm inferential results. Furthermore, through simulation and a real application in political science, we find that matched datasets with high levels of covariate balance tend to approximate balance-constrained designs like rerandomization, and analyzing them as such can lead to precise causal analyses. However, assuming a precise design should be proceeded with caution, because it can harm inferential results if there are still substantial biases due to remaining imbalances after matching. Our approach is implemented in the randChecks R package, available on CRAN.

Testing Biased Randomization Assumptions and Quantifying Imperfect Matching and Residual Confounding in Matched Observational Studies

“…Failure to reject our proposed test, like failure to reject any statistical test, does not translate to a statement about the correctness of the randomization assumption; in fact, statistical matching algorithms are likely to create dependence among matched pairs or sets so that the independence part of the randomization assumption almost surely does not hold. However, through our extensive simulations (see also simulations in Branson (2020)), it seemed that when our proposed tests cannot be rejected, the randomization-based outcome analysis typically has good statistical performance; in other words, the randomization assumption is a good approximation of the complicated reality when it cannot be rejected by our tests.…”

“…In this way, the CPT procedure yields an exact p-value for testing the randomization assumption. Similar Fisherian-permutation-based strategy was also leveraged in Branson (2020) and Branson and Keele (2020) to deliver an exact test for the randomization assumption.…”

On the statistical role of inexact matching in observational studies

“…The balance test did not detect imbalance in any of our simulations. See [7,25,30,10] for further discussion on the role of balance tests.…”

“…However, each of our strata has one untreated unit and (possibly) several treated units while in variable-ratio matching it is the other way round. The stratified experiment reference distribution can also be used for checking covariate balance, in line with the recommendations of Hansen & Bowers [26], Branson [10].…”

“…A prominent tradition in causal inference posits that this problem should be approached using a combination of matching and randomization inference [46,47,49,50,52,19,20,11,10]. The simplest version of this strategy can be described as follows:…”