Assessing robustness against potential publication bias in Activation Likelihood Estimation (ALE) meta-analyses for fMRI

Abstract

The importance of integrating research findings is incontrovertible and procedures for coordinate-based meta-analysis (CBMA) such as Activation Likelihood Estimation (ALE) have become a popular approach to combine results of fMRI studies when only peaks of activation are reported. As meta-analytical findings help building cumulative knowledge and guide future research, not only the quality of such analyses but also the way conclusions are drawn is extremely important. Like classical meta-analyses, coordinate-based meta-analyses can be subject to different forms of publication bias which may impact results and invalidate findings. The file drawer problem refers to the problem where studies fail to get published because they do not obtain anticipated results (e.g. due to lack of statistical significance). To enable assessing the stability of meta-analytical results and determine their robustness against the potential presence of the file drawer problem, we present an algorithm to determine the number of noise studies that can be added to an existing ALE fMRI meta-analysis before spatial convergence of reported activation peaks over studies in specific regions is no longer statistically significant. While methods to gain insight into the validity and limitations of results exist for other coordinate-based meta-analysis toolboxes, such as Galbraith plots for Multilevel Kernel Density Analysis (MKDA) and funnel plots and egger tests for seed-based d mapping, this procedure is the first to assess robustness against potential publication bias for the ALE algorithm. The method assists in interpreting meta-analytical results with the appropriate caution by looking how stable results remain in the presence of unreported information that may differ systematically from the information that is included. At the same time, the procedure provides further insight into the number of studies that drive the meta-analytical results. We illustrate the procedure through an example and test the effect of several parameters through extensive simulations. Code to generate noise studies is made freely available which enables users to easily use the algorithm when interpreting their results.

A meta-analytic approach to mapping co-occurrent grey matter volume increases and decreases in psychiatric disorders

“…We might hypothesize that the dataset of the experiments included in the COA-O analysis could be just a subsample of all the possible evidences about GM changes of opposite polarity, thus biasing our results. To assess their robustness, we implemented a modified version of the Fail-safe technique (Acar et al, 2018). The ratio behind that is that we cannot represent the COA-O network as if we gathered all the possible experiments on the matter, but we still can observe what would happen if our sample was much larger.…”

Widespread and lateralized social brain activity for processing dynamic facial expressions

“…One possible reason for these inconsistent findings is sample size, which was less than 30 in all previous individual experiments with only a few exceptions (Badzakova‐Trajkov et al, ; Kessler et al, ; Rymarczyk et al, ), because studies involving small sample sizes may have low statistical power and a reduced chance of detecting effects (Button et al, ). This issue could be relevant even to meta‐analyses (Arsalidou et al, ; Zinchenko et al, ) because these studies employed coordinate‐based meta‐analytical methods, which assess the convergence of the locations of activation foci reported in individual studies and have difficulty detecting small effects with underpowered individual studies (Acar, Seurinck, Eickhoff, & Moerkerke, ). Another possible reason for these inconsistencies is that a majority of previous studies have compared dynamic with static facial expressions.…”

Invasive Motor Cortex Stimulation Influences Intracerebral Structures in Patients With Neuropathic Pain: An Activation Likelihood Estimation Meta-Analysis of Imaging Data

“…One of the strengths of this review concerns the use of the robust ALE methodology to meta‐analyze imaging results in neuromodulation treatment. Another strength of this paper comprises the homogeneity of the neuroimaging methods used in the included papers.…”