Breaking the circularity in circular analyses: Simulations and formal treatment of the flattened average approach

Abstract

There has been considerable debate and concern as to whether there is a replication crisis in the scientific literature. A likely cause of poor replication is the multiple comparisons problem. An important way in which this problem can manifest in the M/EEG context is through post hoc tailoring of analysis windows (a.k.a. regions-of-interest, ROIs) to landmarks in the collected data. Post hoc tailoring of ROIs is used because it allows researchers to adapt to inter-experiment variability and discover novel differences that fall outside of windows defined by prior precedent, thereby reducing Type II errors. However, this approach can dramatically inflate Type I error rates. One way to avoid this problem is to tailor windows according to a contrast that is orthogonal (strictly parametrically orthogonal) to the contrast being tested. A key approach of this kind is to identify windows on a fully flattened average. On the basis of simulations, this approach has been argued to be safe for post hoc tailoring of analysis windows under many conditions. Here, we present further simulations and mathematical proofs to show exactly why the Fully Flattened Average approach is unbiased, providing a formal grounding to the approach, clarifying the limits of its applicability and resolving published misconceptions about the method. We also provide a statistical power analysis, which shows that, in specific contexts, the fully flattened average approach provides higher statistical power than Fieldtrip cluster inference. This suggests that the Fully Flattened Average approach will enable researchers to identify more effects from their data without incurring an inflation of the false positive rate.

Breakthrough percepts of online identity: Detecting recognition of email addresses on the fringe of awareness

“…The key step in this procedure is the merging of probe and irrelevant. Essentially, as verified in simulation and proof (Bowman et al., 2020), this merging means that, under the null, selecting this window of interest from the resulting aggregated ERP is mathematically orthogonal to the probe minus irrelevant contrast. Accordingly, under the null, there is no increased probability of the probe being different from irrelevant at the window selected on the aggregated ERP.…”

“…Statistical analysis was conducted according to the Aggregated Grand Average of Trials (AGAT) method (Bowman et al., 2020; Brooks et al., 2017). For this, the trials for the probe and irrelevant conditions were merged to provide one aggregated ERP per participant.…”

“…We, then, show evidence of a strong effect at the level of individual participants using our standard permutation procedure (see Method section; Alsufyani et al., 2019; Bowman et al., 2013, 2014). In both analyses, we used our standard Aggregated (Grand or ERP) Average of Trials (see Method section; Bowman et al., 2020; Brooks et al., 2017) as a means to identify analysis windows without inflating false‐positive rates.…”

The missing N1 or jittered P2: Electrophysiological correlates of pattern glare in the time and frequency domain

“…ERPs are typically characterised by a series of positive and negative excursions from baseline, which correspond to the stimulus evoked onset transients, before the time‐series settles back, and we wanted to capture only this period. To do this, we focussed on the aggregated average (Bowman et al, 2020; Brooks et al, 2017) across the three stimulus conditions (for Onsets 2–8). ROIs can be identified on the aggregated average, without inflating false‐positive rates, since it does not reflect condition (i.e.…”

“…ROIs can be identified on the aggregated average, without inflating false‐positive rates, since it does not reflect condition (i.e. stimulus) differences, which for us amounts to the PGI (Bowman et al, 2020; Brooks et al, 2017). However, an initial inspection of our data revealed that the aggregated average did not settle back to baseline but rather fell to a constant, positive DC level.…”