Informed Bayesian Inference for the A/B Test

Gronau, Quentin Frederik; Raj, Akash; Wagenmakers, Eric–Jan

doi:10.18637/jss.v100.i17

Cited by 12 publications

(20 citation statements)

References 33 publications

(41 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A Bayesian reanalysis with a one-sided default alternative hypothesis (i.e., a folded standard normal prior on the log odds ratio) yields a Bayes factor of 4.14 in favor of scriptH+over scriptH0; with equal prior model probability this leaves 19.5% for scriptH0. For details see Gronau et al (in press).…”

mentioning

confidence: 99%

Advantages masquerading as “issues” in Bayesian hypothesis testing: A commentary on Tendeiro and Kiers (2019).

Ravenzwaaij¹,

Wagenmakers²

2022

Psychological Methods

Self Cite

View full text Add to dashboard Cite

Tendeiro and Kiers (2019) provide a detailed and scholarly critique of Null Hypothesis Bayesian Testing (NHBT) and its central component -the Bayes factor-that allows researchers to update knowledge and quantify statistical evidence. Tendeiro and Kiers conclude that NHBT constitutes an improvement over frequentist p-values, but primarily elaborate on a list of eleven 'issues' of NHBT. We believe that several issues identified by Tendeiro and Kiers are of central importance for elucidating the complementary roles of hypothesis testing versus parameter estimation and for appreciating the virtue of statistical thinking over conducting statistical rituals. But although we agree with many of their thoughtful recommendations, we believe that Tendeiro and Kiers are overly pessimistic, and that several of their 'issues' with NHBT may in fact be conceived as pronounced advantages. We illustrate our arguments with simple, concrete examples and end with a critical discussion of one of the recommendations by Tendeiro and Kiers, which is that "estimation of the full posterior distribution offers a more complete picture" than a Bayes factor hypothesis test.

show abstract

mentioning

confidence: 99%

Advantages masquerading as “issues” in Bayesian hypothesis testing: A commentary on Tendeiro and Kiers (2019).

Ravenzwaaij¹,

Wagenmakers²

2022

Psychological Methods

Self Cite

View full text Add to dashboard Cite

show abstract

“…The test proposed by Kass and Vaidyanathan 14 and implemented by Gronau et al 15 does not assign a prior directly to

, but applies a logit transformation and assigns priors to the transformed parameters (

). Specifically, we write:

where

is a grand mean and

is the difference in log odds (ie, the log odds ratio):

…”

Section: Two Ways Of Testing the Equality Of Two Proportionsmentioning

confidence: 99%

“…Such an inverse relation is undesirable in almost all empirical applications, and so values

are therefore to be avoided. Gronau et al 15 developed software to compute the Bayes factor using this prior specification, first proposed by Kass and Vaidyanathan, 14 suggesting

as a default value.…”

Section: Two Ways Of Testing the Equality Of Two Proportionsmentioning

confidence: 99%

See 1 more Smart Citation

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Dablander

Huth

Gronau

et al. 2021

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains, it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis testing by means of the Bayes factor provides one avenue to do so, requiring the specification of prior distributions for parameters. The most popular analysis approach views the comparison of proportions from a contingency table perspective, assigning prior distributions directly to the two proportions. Another, less popular approach views the problem from a logistic regression perspective, assigning prior distributions to logit‐transformed parameters. Reanalyzing 39 null results from the New England Journal of Medicine with both approaches, we find that they can lead to markedly different conclusions, especially when the observed proportions are at the extremes (ie, very low or very high). We explain these stark differences and provide recommendations for researchers interested in testing the equality of two proportions and users of Bayes factors more generally. The test that assigns prior distributions to logit‐transformed parameters creates prior dependence between the two proportions and yields weaker evidence when the observations are at the extremes. When comparing two proportions, we argue that this test should become the new default.

show abstract

“…Two papers in this special issue tackle the problem of hypothesis testing. Gronau, Raj K. N., and Wagenmakers (2021) show how to perform Bayesian inference for A/B tests using R package abtest, which allows for the incorporation of expert knowledge in the priors. Mulder et al (2021) describe a general framework for testing hypotheses using Bayes factors for many different types of models.…”

Section: Hypothesis Testingmentioning

confidence: 99%

Software for Bayesian Statistics

Cameletti

Gómez‐Rubio

2021

J. Stat. Soft.

View full text Add to dashboard Cite

In this summary we introduce the papers published in the special issue on Bayesian statistics. This special issue comprises 20 papers on Bayesian statistics and Bayesian inference on different topics such as general packages for hierarchical linear model fitting, survival models, clinical trials, missing values, time series, hypothesis testing, priors, approximate Bayesian computation, and others.

show abstract

Informed Bayesian Inference for the A/B Test

Cited by 12 publications

References 33 publications

Advantages masquerading as “issues” in Bayesian hypothesis testing: A commentary on Tendeiro and Kiers (2019).

Advantages masquerading as “issues” in Bayesian hypothesis testing: A commentary on Tendeiro and Kiers (2019).

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Software for Bayesian Statistics

Contact Info

Product

Resources

About