A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Abstract

Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains, it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis testing by means of the Bayes factor provides one avenue to do so, requiring the specification of prior distributions for parameters. The most popular analysis approach views the comparison of proportions from a contingency table perspective, assigning prior distributions directly to the two proportions. Another, less popular approach views the problem from a logistic regression perspective, assigning prior distributions to logit‐transformed parameters. Reanalyzing 39 null results from the New England Journal of Medicine with both approaches, we find that they can lead to markedly different conclusions, especially when the observed proportions are at the extremes (ie, very low or very high). We explain these stark differences and provide recommendations for researchers interested in testing the equality of two proportions and users of Bayes factors more generally. The test that assigns prior distributions to logit‐transformed parameters creates prior dependence between the two proportions and yields weaker evidence when the observations are at the extremes. When comparing two proportions, we argue that this test should become the new default.

Multicentre randomised trials in anaesthesia: an analysis using Bayesian metrics

“…It is important to recognise that the post‐test probabilities can be calculated using pre‐test probabilities other than 0.5 (see Appendix 1 in reference [9]). Furthermore, several approaches may be used to calculate a Bayes factor for the equivalence of two proportions, including analytic methods [12, 21], as used here, and by simulated statistical testing [11, 13]. Different methods yield different values for the Bayes factor, although in our experience not dramatically so.…”

A Bayesian One-Sample Test for Proportion

“…In this case, the bias against the null hypothesis is 0.1182. On the other hand, the bias in favor of the null hypothesis is measured by computing (5) with θ 0 = 0.1 ± 0.05, which gives 0.0504 (for θ 0 = 0.15) and 0.0204 (for θ 0 = 0.05). This shows an acceptable bias either for or against the null hypothesis for the uniform [0, 1] prior.…”

“…Furthermore, ref. [5] provided some recommendations for using Bayes factors in testing the equality of two proportions to improve the sensitivity of the test. Several studies can also be found in the literature on developing the Bayesian two-sample proportion test in contingency tables for testing the independence between rows and columns.…”

Bayesian tests of two proportions: A tutorial with R and JASP

“…The two approaches make different assumptions, ask different questions, and therefore provide different answers (cf. Dablander et al, 2022).…”

“…This article is a tutorial, and consequently we will emphasize the assumptions, interpretations, and practical application of the test. A more advanced discussion of the software can be found in Gronau et al (2021), and an associated statistical paradox is presented in Dablander, Huth, Gronau, Etz, and Wagenmakers (2022).…”