A permutation‐based approach for heterogeneous meta‐analyses of rare events

Abstract

The increasingly widespread use of meta-analysis has led to growing interest in meta-analytic methods for rare events and sparse data. Conventional approaches tend to perform very poorly in such settings. Recent work in this area has provided options for sparse data, but these are still often hampered when heterogeneity across the available studies differs based on treatment group. We propose a permutation-based approach based on conditional logistic regression that accommodates this common contingency, providing more reliable statistical tests when such patterns of heterogeneity are observed. We find that commonly used methods can yield highly inflated Type I error rates, low confidence interval coverage, and bias when events are rare and non-negligible heterogeneity is present. Our method often produces much lower Type I error rates and higher confidence interval coverage than traditional methods in these circumstances.We illustrate the utility of our method by comparing it to several other methods via a simulation study and analyzing an example data set, which assess the use of antibiotics to prevent acute rheumatic fever.

Exact inference for the random‐effect model for meta‐analyses with rare events

“…As noted by Zabriskie et al, the current setting has been largely underappreciated in the meta-analysis literature. The Cochrane Handbook for Systematic Reviews of Interventions suggests that "incorporation of heterogeneity into an estimate of a treatment effect should be a secondary consideration when attempting to produce estimates of effects from sparse data -the primary concern is to discern whether there is any signal of an effect in the data" 30,36 . Our numerical studies, however, illustrate that the presence of heterogeneity has a huge bearing on the conclusions drawn from meta-analyses in such settings.…”

“…While these methods all target the setting with low event rates, the proposed inference procedures remain asymptotic in the number of studies and, to the best of our knowledge, are not available in open-source software. Most recently, Zabriskie et al proposed a permutation-based approach based on conditional logistic regression 30 . However, this method cannot be applied to all data sets, does not uniformly guarantee type I error control, and is computationally intensive.…”