On p-Values and Bayes Factors

Abstract

The p-value quantifies the discrepancy between the data and a null hypothesis of interest, usually the assumption of no difference or no effect. A Bayesian approach allows the calibration of p-values by transforming them to direct measures of the evidence against the null hypothesis, so-called Bayes factors. We review the available literature in this area and consider two-sided significance tests for a point null hypothesis in more detail. We distinguish simple from local alternative hypotheses and contrast traditional Bayes factors based on the data with Bayes factors based on p-values or test statistics. A well-known finding is that the minimum Bayes factor, the smallest possible Bayes factor within a certain class of alternative hypotheses, provides less evidence against the null hypothesis than the corresponding p-value might suggest. It is less known that the relationship between p-values and minimum Bayes factors also depends on the sample size and on the dimension of the parameter of interest. We illustrate the transformation of p-values to minimum Bayes factors with two examples from clinical research.

Bayesian Calibration ofp‐Values from Fisher's Exact Test

“…Another calibration, which is directly based on the two‐sided p ‐value p and can be derived along similar lines as the ‘ ’ calibration (Held & Ott, , section 2.3), is This bound is obtained by replacing p in by q = 1 − p and will thus be called the ‘ ’ calibration in the sequel. Note that calibration is a much lower bound than the minimum test‐based Bayes factors and .…”

“…Note that calibration is a much lower bound than the minimum test‐based Bayes factors and . In the linear model, calibration has been shown (Held & Ott, ) to be a lower bound on the sample‐size adjusted minBF based on the g ‐prior for any sample size n ≥ d + 3, where d is the number of covariates in the model (standard regularity conditions require n ≥ d + 2). In fact, for d = n + 3, these minBFs converge from above to the bound as d → ∞ , hence the subscript ∞ in the notation above.…”

“…To obtain the corresponding minBF for a two‐sided p ‐value p , it suffices to consider the class of all two‐point distributions symmetric around the null value. The minBF then turns out to be (Berger & Sellke, ; Held & Ott, ) where φ (.) denotes the standard normal density function and t ⋆ = Φ −1 (1 − p /2) = | t |.…”

“…It is a well-known finding that for point null hypotheses, minimum Bayes factors provide less evidence against H 0 than the corresponding p-value might suggest (Berger & Sellke, 1987;Sellke et al, 2001). Ghosh et al (2005) and Held & Ott (2018) review the literature on minBFs and transformations of p-values to minBFs.…”

Sharpen statistical significance: Evidence thresholds and Bayes factors sharpened into Occam's razor

“…Such calibrations not only support the intuition of Fraser, Reid, and Wong () and others that an extremely low p value indicates strong evidence against the null hypothesis but also support arguments against considering p values near 0.05 as indicative of strong evidence (e.g., Sellke, Bayarri, & Berger, ). See Held and Ott () for a recent review.…”