Bias caused by sampling error in meta-analysis with small sample sizes

Abstract

BackgroundMeta-analyses frequently include studies with small sample sizes. Researchers usually fail to account for sampling error in the reported within-study variances; they model the observed study-specific effect sizes with the within-study variances and treat these sample variances as if they were the true variances. However, this sampling error may be influential when sample sizes are small. This article illustrates that the sampling error may lead to substantial bias in meta-analysis results.MethodsWe conducted extensive simulation studies to assess the bias caused by sampling error. Meta-analyses with continuous and binary outcomes were simulated with various ranges of sample size and extents of heterogeneity. We evaluated the bias and the confidence interval coverage for five commonly-used effect sizes (i.e., the mean difference, standardized mean difference, odds ratio, risk ratio, and risk difference).ResultsSampling error did not cause noticeable bias when the effect size was the mean difference, but the standardized mean difference, odds ratio, risk ratio, and risk difference suffered from this bias to different extents. The bias in the estimated overall odds ratio and risk ratio was noticeable even when each individual study had more than 50 samples under some settings. Also, Hedges’ g, which is a bias-corrected estimate of the standardized mean difference within studies, might lead to larger bias than Cohen’s d in meta-analysis results.ConclusionsCautions are needed to perform meta-analyses with small sample sizes. The reported within-study variances may not be simply treated as the true variances, and their sampling error should be fully considered in such meta-analyses.

Arcsine‐based transformations for meta‐analysis of proportions: Pros, cons, and alternatives

“…Because conventional meta-analysis models assume normally distributed data, 11 especially when the sample sizes are small. [23][24][25][26][27] In addition, in the presence of zero event counts, both log-and logit-transformed proportions cannot be calculated, and a continuity correction must be applied to the zero counts, usually by adding 0.5. [28][29][30] This correction may have considerable impact on the synthesized proportion for rare events.…”

Comparison of four heterogeneity measures for meta‐analysis

“…Third, as in conventional meta‐analysis methods, the within‐study variances were treated as true values in our analysis and the four heterogeneity measures. However, they were actually estimates and were subject to sampling error; these variance estimates could be poor if the corresponding studies had small sample sizes . Future studies are highly needed to effectively account for such sampling errors in within‐study variances and to quantify heterogeneity more accurately.…”

The magnitude of small-study effects in theCochrane Database of Systematic Reviews: an empirical study of nearly 30 000 meta-analyses

“…Also, Egger’s regression may have inflated false-positive rates in certain cases 17. The inflation is due to the intrinsic association between effect sizes and their within-study variances; such effect sizes include ORs, risk ratios, risk differences, as well as standardised mean differences 17 32–37. These issues need to be carefully taken into account when exploring publication bias.…”