Sensitivity analysis for publication bias in meta-analyses

Abstract

We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that "statistically significant" results are more likely to be published than negative or "nonsignificant" results by an unknown ratio, eta. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse-probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as: "For publication bias to shift the observed point estimate to the null, 'significant' results would need to be at least 30-fold more likely to be published than negative or 'nonsignificant' results." Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of eta across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and "nonsignificant" studies; this method sometimes indicates that no amount of such publication bias could "explain away" the results. We illustrate the proposed methods using real-life meta-analyses and provide an R package, PublicationBias.

Estimating publication bias in meta‐analyses of peer‐reviewed studies: A meta‐meta‐analysis across disciplines and journal tiers

“…The sensitivity analysis methods assess: (1) the minimum selection ratio that would be required to attenuate a meta-analytic pooled point estimate to the null; and (2) the minimum selection ratio that would be required to shift the confidence interval to include the null. They also allow estimation of a "worst-case" pooled point estimate and confidence interval under maximal publication bias in which affirmative studies are almost infinitely more likely to be published than nonaffirmative studies; these worst-case estimates are obtained by simply meta-analyzing only the nonaffirmative studies 49 . These methods obviate the distributional and independence assumptions required for our main analysis models, providing a form of sensitivity analysis for the main results.…”

Reducing meat consumption by appealing to animal welfare: protocol for a meta-analysis and theoretical review

“…We will conduct several sensitivity analyses regarding statistical biases and the scope of articles included in the analysis. First, we will assess publication bias using selection model methods [46], sensitivity analysis methods [47], and the significance funnel plot [47]. Given the relatively small anticipated number of eligible studies, we anticipate that selection models may fail to converge or may provide extremely wide, uninformative confidence intervals 2 , in which case we may omit them and present only the sensitivity analysis approach.…”

Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

“…these worst-case estimates are obtained by simply meta-analyzing only the nonaffirmative studies (40). These methods obviate the distributional and independence assumptions required for our main analysis models, providing a form of sensitivity analysis for the main results.…”

“…For these analyses, we retained all point estimates from each meta-analysis and used a robust sensitivity analysis model to account for clustering and non-normality (40)…”