Behavior, sensitivity, and power of activation likelihood estimation characterized by massive empirical simulation

Abstract

Given the increasing number of neuroimaging publications, the automated knowledge extraction on brain-behavior associations by quantitative meta-analyses has become a highly important and rapidly growing field of research. Among several methods to perform coordinate-based neuroimaging meta-analyses, Activation Likelihood Estimation (ALE) has been widely adopted. In this paper, we addressed two pressing questions related to ALE meta-analysis: i) Which thresholding method is most appropriate to perform statistical inference? ii) Which sample size, i.e., number of experiments, is needed to perform robust meta-analyses? We provided quantitative answers to these questions by simulating more than 120,000 meta-analysis datasets using empirical parameters (i.e., number of subjects, number of reported foci, distribution of activation foci) derived from the BrainMap database. This allowed to characterize the behavior of ALE analyses, to derive first power estimates for neuroimaging meta-analyses, and to thus formulate recommendations for future ALE studies. We could show as a first consequence that cluster-level family-wise error (FWE) correction represents the most appropriate method for statistical inference, while voxel-level FWE correction is valid but more conservative. In contrast, uncorrected inference and false-discovery rate correction should be avoided. As a second consequence, researchers should aim to include at least 20 experiments into an ALE meta-analysis to achieve sufficient power for moderate effects. We would like to note, though, that these calculations and recommendations are specific to ALE and may not be extrapolated to other approaches for (neuroimaging) meta-analysis.

O3‐08‐02: Resting‐State Network Dysfunction in Alzheimer's Disease: A Systematic Review And Meta‐Analysis

“…Nonparametric P values were assessed at a familywise error-corrected threshold of P , .05 on a cluster level (cluster-forming threshold: P ,.001 at voxel level) and transformed into t scores for display purposes. Only contrasts including more than 18 experiments were considered, as recommended in a recent large-scale simulation study [20].…”

Stuttering as a trait or a state revisited: motor system involvement in persistent developmental stuttering

“…These errors affected published ALE analyses using False-Discovery Rate (FDR) corrections for multiple comparisons prior to May 11, 2015 (GINGERALE versions prior to v.2.3.3) and cluster-wise Family-Wise Error (cFWE) corrections for multiple comparisons prior to April 26, 2016 (GINGERALE versions prior to v2.3.6). The implementation errors in these versions may have caused statistical thresholds in the resultant ALE analyses to be more liberal than intended by the researchers, including in our own analysis (Belyk et al, 2015).Furthermore, subsequent research has demonstrated that voxel-wise FDR correction in the context of ALE has the undesirable properties of being simultaneously low in sensitivity to true effects and highly susceptible to false positives (Eickhoff et al, 2016). This view is supported by a broader theoretical position that voxel-wise FDR may be inappropriate for spatially smooth data, such as the data represented in ALE analyses (Chumbley & Friston, 2009).…”

“…Furthermore, subsequent research has demonstrated that voxel-wise FDR correction in the context of ALE has the undesirable properties of being simultaneously low in sensitivity to true effects and highly susceptible to false positives (Eickhoff et al, 2016). This view is supported by a broader theoretical position that voxel-wise FDR may be inappropriate for spatially smooth data, such as the data represented in ALE analyses (Chumbley & Friston, 2009).…”

Coordinate based meta-analysis does not show grey matter atrophy in narcolepsy

“…Eight (seven without the ROI study) studies are intuitively too few for reasonable statistical power, but the concept of power in CBMA is complicated by its explicit dependency on the specifics of the contributing studies. Nevertheless, a recent quantitative analysis of statistical power of the ALE algorithm [14] concluded that 20 studies are a minimum for typical effects (around 25% of studies reporting the effect), and that with just 7 valid studies the estimated power is only around 20%. Indeed, with 7 studies the statistical power is such that only strong effects (most studies reporting coordinates in the same anatomical location) would be detectible by ALE, and these would be apparent by direct inspection of the studies.…”

“…These results were subsequently demonstrated to be due to an implementation issue with the software [11], and re-analysis using a corrected version (GingerALE 2.3.3) detected no consistent significant regional GM loss [12]. This reanalyses employed the false discovery rate (FDR) method of controlling the type 1 error [13], which is no longer the recommended option in GingerALE [14] and has been superseded [15]. This has prompted a second re-evaluation of the narcolepsy data by Zhong and colleagues [16] employing the signed differential mapping (SDM) CBMA algorithm [17,18].…”