Cluster wild bootstrapping to handle dependent effect sizes in meta‐analysis with a small number of studies

Abstract

The most common and well-known meta-regression models work under the assumption that there is only one effect size estimate per study and that the estimates are independent. However, meta-analytic reviews of social science research often include multiple effect size estimates per primary study, leading to dependence in the estimates. Some meta-analyses also include multiple studies conducted by the same lab or investigator, creating another potential source of dependence. An increasingly popular method to handle dependence is robust variance estimation (RVE), but this method can result in inflated Type I error rates when the number of studies is small. Small-sample correction methods for RVE have been shown to control Type I error rates adequately but may be overly conservative, especially for tests of multiplecontrast hypotheses. We evaluated an alternative method for handling dependence, cluster wild bootstrapping, which has been examined in the econometrics literature but not in the context of meta-analysis. Results from two simulation studies indicate that cluster wild bootstrapping maintains adequate Type I error rates and provides more power than extant small-sample correction methods, particularly for multiplecontrast hypothesis tests. We recommend using cluster wild bootstrapping to conduct hypothesis tests for meta-analyses with a small number of studies. We have also created an R package that implements such tests.

Minority Stress and Positive Identity Aspects in Members of LGBTQ+ Parent Families: Literature Review and a Study Protocol for a Mixed-Methods Evidence Synthesis

“…We will obtain Wald-type confidence intervals for the summary effect and profile likelihood confidence intervals for τ 2 . Additionally, we will use cluster wild bootstrapping for hypothesis tests in meta-regression models [ 134 ].…”

Minority stress and positive identity aspects in members of LGBTQ+ parent families: Study protocol for a mixed-methods systematic review and meta-analysis

“…We will obtain Wald-type confidence intervals for the summary effect and profile likelihood confidence intervals for τ 2 . Additionally, we will use cluster wild bootstrapping for hypothesis tests in meta-regression models (Joshi et al, 2022).…”

Advanced methods and implementations for the meta-analyses of animal models: Current practices and future recommendations

“…Together with the estimated variance components (e.g., and ) and sampling variance (i.e., ), the variance (and covariance if involving correlated ) matrix can be constructed and model coefficients can be estimated under the inverse-variance permutation test and bootstrapping) or cluster-robust inference (sandwich-type estimator; see section 9.3). The use of methods based on t-distribution (with adjusted degrees of freedom), permutation test and cluster-robust inference are preferable in the case of a meta-analysis with a small number of studies (Joshi et al, 2022;Nakagawa et al, 2021c;Sánchez-Meca and Marín-Martínez, 2008;Viechtbauer et al, 2015). We illustrate how to implement multilevel models with recommended estimators (REML) and improved inference methods (t-distribution with adjusted degrees of freedoms) and interpret corresponding model results using rma.mv() function in metafor package in section 11 (Supplementary Materials file 2;…”

“…Several small sample-size adjustment methods can be used to address these issues (Pustejovsky and Tipton, 2018;Tipton and Pustejovsky, 2015;Welz et al, 2023). Recently, the robust-wild-bootstrapping method has been introduced to control Type I error rates while improving the statistical power of hypothesis tests in RVE (Joshi et al, 2022).…”