An Application of Balanced Repeated Replication (Brr) Variance Estimation To Program Evaluation

Abstract

Program evaluations often are based on data drawn from samples of observation units that are not simple random samples. In particular, samples often may be clustered to maximize the number of observations that can be obtained, given project cost constraints. However, clustering gives rise to well-known problems in estimating the variance of sample estimates. The most common procedure is to apply a scalar "design effect" multiplier more or less judgmentally to inflate sample variances. An alternative procedure, as reported in this article, is to construct balanced repeated replication (BRR) estimates of sample variances. BRR estimates of variance are constructed from orthogonally-weighted subsamples of the data, and can provide better estimates of variances from complex samples than other methods.

A primer on synthesizing individual participant data obtained from complex sampling surveys: A two-stage IPD meta-analysis approach.

“…The design-based approach corrects for the complex sampling design by reestimating sampling variances and standard errors in models without multiple analytic levels. Taylor series linearization (Lumley, 2004), replication methods (Rust, 2013), normalized weights (Cavin & Ohls, 1990), test statistics adjustments (Thomas & Heck, 2001), and effect size corrections (Hedges, 2007) are some strategies that can be used to account for the dependencies of the data using a design-based strategy. The model-based approach partitions the variances in the variables of interest into their components at multiple analytic levels.…”

A Primer on Synthesizing Individual Participant Data Obtained From Complex Sampling Surveys: A Two-Stage IPD Meta-Analysis Approach

“…The design-based approach corrects for the complex sampling design by re-estimating sampling variances and standard errors in models without multiple analytical levels. Taylor series linearization (Lumley, 2004), replication methods (Rust, 2013), normalized weights (Cavin & Ohls, 1990), test statistics adjustments (Thomas & Heck, 2001), and effect size corrections (Hedges, 2007) are some strategies that can be used to account for the dependencies of the data using a design-based strategy. The model-based approach partitions the variances in the variables of interest into their components at multiple analytical levels.…”

Methodological Considerations in Using Complex Survey Data

“…Also, there is usually some level of unit nonresponse in complex survey data (Pike 2008;Lee, Forthofer, and Lorimor 1989). This type of complex sampling design is much more efficient than a random sample (Cavin and Ohls 1990), but it comes at the expense of requiring the researcher to employ analytic strategies in their computation to ensure that the results are representative of the intended population and the variance estimates are correct.…”

“…In BRR methods, each stratum is divided into one half and the estimate of interest is computed from this half sample (i.e., a replicate). The design-corrected variance can then be estimated on the basis of the estimates from the half samples using the full-sample mean (Cavin and Ohls 1990). BRR is restricted to cases where there are only two PSUs selected from each stratum (Rust and Rao 1996).…”

“…Second design-based method: Apply a normalized weight adjusted by the design effect. When specialized software is not available, a second designbased approach for addressing disproportionate sampling and nonindependence is to adjust the normalized weight by the design effect (DEFF; Cavin and Ohls 1990). A normalized weight is computed as the raw weight divided by its mean (Thomas and Heck 2001;Thomas, Heck, and Bauer 2005;Peng 2000).…”