The look-elsewhere effect from a unified Bayesian and frequentist perspective

Abstract

When searching over a large parameter space for anomalies such as events, peaks, objects, or particles, there is a large probability that spurious signals with seemingly high significance will be found. This is known as the look-elsewhere effect and is prevalent throughout cosmology, (astro)particle physics, and beyond. To avoid making false claims of detection, one must account for this effect when assigning the statistical significance of an anomaly. This is typically accomplished by considering the trials factor, which is generally computed numerically via potentially expensive simulations. In this paper we develop a continuous generalization of the Bonferroni and Šidák corrections by applying the Laplace approximation to evaluate the Bayes factor, and in turn relating the trials factor to the prior-to-posterior volume ratio. We use this to define a test statistic whose frequentist properties have a simple interpretation in terms of the global p-value, or statistical significance. We apply this method to various physics-based examples and show it to work well for the full range of p-values, i.e. in both the asymptotic and non-asymptotic regimes. We also show that this method naturally accounts for other model complexities such as additional degrees of freedom, generalizing Wilks' theorem. This provides a fast way to quantify statistical significance in light of the look-elsewhere effect, without resorting to expensive simulations.

Challenges for $Λ$CDM: An update

“…However, this argument may not be applicable when the considered statistics are simple and generic as are most of the signals discussed below (for a detailed discussion, see Refs. [1008][1009][1010]).…”

On the absence of backsplash analogues to NGC 3109 in the ΛCDM framework

“…These probabilities are a frequentist interpretation of the "number of trials" for each individual test (e.g. Bayer & Seljak 2020). Since the timing argument analysis of considered the kinematics of 33 LG non-satellite dwarf galaxies of which NGC 3109 was not the only HVG, the challenge to ΛCDM posed by NGC 3109 is difficult to understand merely via the look-elsewhere effect.…”

Detecting the periodicity of highly irregularly sampled light-curves with Gaussian processes: the case of SDSSJ025214.67-002813.7

“…In any case, the source considered in Liao et al (2021) and in this study was singled out by a sample of 625 analysed objects (Chen et al 2020). This implies the reported probabilities have to be corrected for the so-called "look elsewhere effect" (Bayer & Seljak 2020), i.e., the probability to find a positive result across multiple (independent) trials. In our case the ∼ 4.2𝜎 probability supporting the periodic kernel becomes, after the multiple trial correction (van der Klis 1989), ∼ 2.4𝜎.…”