Variability of Bayes Factor estimates in Bayesian Analysis of Variance

Pfister, Roland

doi:10.20982/tqmp.17.1.p040

Cited by 8 publications

(4 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In simulation conditions with a true population effect we added data sampled from a normal distribution with SD = 1, and M = 0.5 or M = 1.25, thus implementing an interaction effect (cf. Pfister, 2021). This also roughly corresponded to effect sizes of η 2 G = 0, η 2 G = .02, and η 2 G = .08, respectively.…”

Section: Methodssupporting

confidence: 56%

A Cautionary Note Against Selective Applications of the Bayes Factor

Schreiner,

Kunde

2024

Preprint

View full text Add to dashboard Cite

Bayes-factor analysis becomes increasingly popular, among other reasons, because it allows to provide evidence for the null hypothesis which is not easily possible with the traditional frequentist approach. A conceivable strategy that apparently takes favorable aspects of both approaches on board, is to use traditional frequentist analyses first, and to support theoretically interesting nil effects by Bayesian analyses thereafter. Here we asked whether such a selective application of Bayesian analyses to only non-significant effects of foregoing frequentist analyses creates bias. In two simulation studies we observed that such selective application of Bayesian analyses in fact severely overestimates evidence in favor of the null hypotheses, when a true population effect exists. While this bias can be attenuated by using more informative priors in the Bayesian analyses, we recommend to not apply such selective combination of analytical approaches but instead to use either frequentist or Bayesian analyses consistently.

show abstract

Section: Methodssupporting

confidence: 56%

A Cautionary Note Against Selective Applications of the Bayes Factor

Schreiner,

Kunde

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…If any of the three ANOVAs returned a non-significant effect of accuracy, we would compute Bayes factors for the pairwise comparison of RDs between correct and erroneous responses and collect additional data in increments of two participants until reaching a Bayes factor of BF 01 > 10 or BF 01 < 0.10 for all three analyses (using a Cauchy distribution with a scale parameter of 1 as prior) or until reaching a sample of 100 analysable datasets. We used separate Bayesian t -tests rather than Bayesian ANOVA, to avoid the variability of Bayes factor estimates inherent in current approaches to the latter [ 34 ].…”

Section: Methodsmentioning

confidence: 99%

Error cancellation

et al. 2022

Self Cite

View full text Add to dashboard Cite

The human cognitive system houses efficient mechanisms to monitor ongoing actions. Upon detecting an erroneous course of action, these mechanisms are commonly assumed to adjust cognitive processing to mitigate the error's consequences and to prevent future action slips. Here, we demonstrate that error detection has far earlier consequences by feeding back directly onto ongoing motor activity, thus cancelling erroneous movements immediately. We tested this prediction of immediate auto-correction by analysing how the force of correct and erroneous keypress actions evolves over time while controlling for cognitive and biomechanical constraints relating to response time and the peak force of a movement. We conclude that the force profiles are indicative of active cancellation by showing indications of shorter response durations for errors already within the first 100 ms, i.e. between the onset and the peak of the response, a timescale that has previously been related solely to error detection. This effect increased in a late phase of responding, i.e. after response force peaked until its offset, further corroborating that it indeed reflects cancellation efforts instead of consequences of planning or initiating the error.

show abstract

“…A practical consequence of using the MRE- and SFR-model specifications is that the added random slopes greatly increase the number of parameters and make the models more challenging to fit. This leads not only to longer computation times but also more variation in the Bayes factors (Pfister, 2021). If the computation time becomes infeasible, we recommend to first explore the model space using a Laplace approximation.…”

Section: Example Data: Stroop Effectmentioning

confidence: 99%

Bayesian Repeated-Measures Analysis of Variance: An Updated Methodology Implemented in JASP

Bergh

Wagenmakers

Aust

2023

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

Analysis of variance (ANOVA) is widely used to assess the influence of one or more experimental (or quasi-experimental) manipulations on a continuous outcome. Traditionally, ANOVA is carried out in a frequentist manner using p values, but a Bayesian alternative has been proposed. Assuming that the proposed Bayesian ANOVA is closely modeled after its frequentist counterpart, one may be surprised to find that the two can yield very different conclusions when the design involves multiple repeated-measures factors. We illustrate such a discrepancy with a real data set from a two-factorial within-subject experiment. For this data set, the results of a frequentist and Bayesian ANOVA are in a disagreement about which main effect accounts for the variance in the data. The reason for this disagreement is that frequentist and the proposed Bayesian ANOVA use different model specifications. As currently implemented, the proposed Bayesian ANOVA assumes that there are no individual differences in the magnitude of effects. We suspect that this assumption is neither obvious to nor desired by most analysts because it is untenable in most applications. We argue here that the Bayesian ANOVA should be revised to allow for individual differences. As a default, we suggest the standard frequentist model specification but discuss a recently proposed alternative and provide guidance on how to choose the appropriate model specification. We end by discussing the implications of the revised model specification for previously published results of Bayesian ANOVAs.

show abstract

Variability of Bayes Factor estimates in Bayesian Analysis of Variance

Cited by 8 publications

References 16 publications

A Cautionary Note Against Selective Applications of the Bayes Factor

A Cautionary Note Against Selective Applications of the Bayes Factor

Error cancellation

Bayesian Repeated-Measures Analysis of Variance: An Updated Methodology Implemented in JASP

Contact Info

Product

Resources

About