(Nearly) optimal P values for all Bell inequalities

Abstract

A key objective in conducting a Bell test is to quantify the statistical evidence against a local-hidden variable model (LHVM) given that we can collect only a finite number of trials in any experiment. The notion of statistical evidence is thereby formulated in the framework of hypothesis testing, where the null hypothesis is that the experiment can be described by an LHVM. The statistical confidence with which the null hypothesis of an LHVM is rejected is quantified by the so-called P value, where a smaller P value implies higher confidence. Establishing good statistical evidence is especially challenging if the number of trials is small, or the Bell violation very low. Here, we derive the optimal P value for a large class of Bell inequalities. What is more, we obtain very sharp upper bounds on the P value for all Bell inequalities. These values are easily computed from the experimental data, and are valid even if we allow arbitrary memory in the devices. Our analysis is able to deal with imperfect random number generators, and event-ready schemes, even if such a scheme can create different kinds of entangled states. Finally, we review requirements for sound data collection, and a method for combining P values of independent experiments. The methods discussed here are not specific to Bell inequalities. For instance, they can also be applied to the study of certified randomness or to tests of noncontextuality.

Test of the no‐signaling principle in the Hensen loophole‐free CHSH experiment

“…This rather peculiar figure does not suggest that it was decided before the start of the experiment, nor is there any indication in the text that it was. This is quite problematic since the theoretical work that supports the derivation of the expression of the p-values used by Hensen et al demands that the total number of trials is fixed before the experiment is started [18,16]. Similarly, we need to be careful when we change the event-ready sample selected for analysis in a test of the no-signalling principle.…”

“…In this respect, it should be pointed out that with the sophisticated method that makes no assumptions on the memory of the devices [16,18], the best characterisation of the p-value is that it is nevertheless bounded from above by the probability that one would obtain assuming i.i.d. directly.…”

“…One essential principle in statistics that is particularly relevant in this context is that the rules establishing how the data is collected and how the statistical analysis is performed should be decided independently of the data [18]. Failure to do so leaves open the possibility to tweak the acquisition or the statistical analysis towards a preferred outcome (the expectation bias [12]).…”

“…The best way to enforce this principle is to establish those rules before the data collection starts [18], and to strictly adhere to them. Surprisingly, the analysis performed by Hensen et al [13] may have fallen victim of this pitfall.…”

“…One parameter in particular that should be decided independently of the data is the number of trials to be collected. If the number of trials is not fixed beforehand, then one can be tempted to record trials until the p-value reaches desirably low limit: an incorrect practice known as p-value fishing [18,12].…”