Using Monte Carlo experiments to select meta‐analytic estimators

Abstract

The purpose of this study is to show how Monte Carlo analysis of meta-analytic estimators can be used to select estimators for specific research situations. Our analysis conducts 1,620 individual experiments, where each experiment is defined by a unique combination of sample size, effect heterogeneity, effect size, publication selection mechanism, and other research characteristics. We compare eleven estimators commonly used in medicine, psychology, and the social sciences. These are evaluated on the basis of bias, mean squared error (MSE), and coverage rates. For our experimental design, we reproduce simulation environments from four recent studies: Stanley, Doucouliagos, & Ioannidis (2017), Alinaghi & Reed (2018), Bom & Rachinger (2019), and Carter et al. (2019a). We demonstrate that relative estimator performance differs across performance measures. An estimator that may be especially good with respect to MSE may perform relatively poorly with respect to coverage rates. We also show that sample size and effect heterogeneity are important determinants of relative estimator performance. We use these results to demonstrate how the observable characteristics of sample size and effect heterogeneity can guide the meta-analyst in choosing the estimators most appropriate for their research circumstances. All of the programming code and output files associated with this project are available at https://osf.io/pr4mb/.

Taxes and Economic Growth in OECD Countries: A Meta-analysis

“…While the FAT procedure that we employ is the predominant approach in economics, methods to correct publication bias remains an active research area. Hong and Reed (2020) demonstrate that current methods, including FAT, often are not effective at eliminating bias. 20.…”

Neglect of publication bias compromises meta-analyses of educational research

“…A large majority of these meta-analyses employed only the trim & fill method for bias correction. Under publication bias, this method is, however, known to suffer from a consistent upward bias and a high false-positive rate similar to that of a naïve meta-analytic estimate [ 14 , 15 , 37 ]. Although better-performing techniques such as the selection models (see [ 7 ]) have been around for many years now, only 3% of reviewed meta-analyses used them.…”

“…In this respect, Friese and Frankenbach have shown that the effect of publication bias gets amplified if some of the null or negative results are transformed into positive results–especially under moderate publication bias and small true effects [ 45 ]. Lastly, there is considerable inherent uncertainty associated with the adjusted effect size estimates [ 14 , 15 , 37 , 46 ], also leading to lower statistical power for detecting more subtle effect sizes.…”

“…There are also several other approaches to publication bias correction [see 15 ] and new ones are continually being developed. In many practical applications, it may be sensible to supplement selection models with different approaches.…”

“…Simulations also show that under even medium publication bias, naïve meta-analytic estimates unadjusted for publication bias show a substantial upward bias and the false-positive rate approaches 100% with the increasing number of included effects [ 14 , 15 ]. Instead of factoring out study idiosyncrasies and protecting the consumers of research from single-study biases, an unadjusted meta-analysis (under publication bias) tends to find seemingly precisely estimated, but substantially inflated effects.…”