Exploring consequences of simulation design for apparent performance of methods of meta-analysis

Abstract

Contemporary statistical publications rely on simulation to evaluate performance of new methods and compare them with established methods. In the context of random-effects meta-analysis of log-odds-ratios, we investigate how choices in generating data affect such conclusions. The choices we study include the overall log-odds-ratio, the distribution of probabilities in the control arm, and the distribution of study-level sample sizes. We retain the customary normal distribution of study-level effects. To examine the impact of the components of simulations, we assess the performance of the best available inverse–variance–weighted two-stage method, a two-stage method with constant sample-size-based weights, and two generalized linear mixed models. The results show no important differences between fixed and random sample sizes. In contrast, we found differences among data-generation models in estimation of heterogeneity variance and overall log-odds-ratio. This sensitivity to design poses challenges for use of simulation in choosing methods of meta-analysis.

On the Q statistic with constant weights in meta-analysis of binary outcomes

“…One could consider assessing heterogeneity by estimating the parameter for the between-study variance in a mixed-effects logistic regression model. That approach, however, would assume likelihoods that Q F does not need, and results in [28] and [29] show that it does not work well.…”

Random‐effects meta‐analysis models for the odds ratio in the case of rare events under different data‐generating models: A simulation study

“…While the performance of BinFI2 and BinRI2 for different DGMs was investigated in the simulation study by Kulinskaya et al. (2021), a similar investigation of BinFI and BinRI is still pending. We outline our expectations with regard to the performance of binomial‐normal GLMMs under different DGMs in the third section.…”

“…However, continuity corrections do not resolve all problems of the inverse variance model in rare events meta‐analyses: Concerns with regard to the fulfillment of distributional assumptions and biases induced by the correlation between $$\hat{\theta }_i$$ and $$\hat{w}_i$$ remain (Jackson & White, 2018; Kulinskaya et al., 2021). Thus, it has been proposed to avoid continuity corrections altogether (Kuss, 2014) and instead use statistical models that are naturally able to incorporate the evidence from zero studies.…”

“…A recent simulation study by Kulinskaya et al. (2021) explored the consequences of using different DGMs when assessing the performance of the inverse variance model and different variants of the binomial‐normal GLMM. Their results clearly show that in addition to evaluating performance for a wide range of parameter values, it is at least equally important to assess a model's robustness to the DGM.…”

A comparison of confidence distribution approaches for rare event meta‐analysis

“…Future work could include examining these CD method under a variety of data generating assumptions and models. Kulinskaya et al 64 illustrate how these choices can impact meta‐analysis results, and Pateras et al 53 note that this impact is especially great for meta‐analyses of studies with small sample sizes. We suspect small study meta‐analysis coupled with rare events would result in even greater distinctions between the data generating methods.…”