A Battle in the Statistics Wars: a simulation-based comparison of Bayesian, Frequentist and Williamsonian methodologies

Abstract

The debates between Bayesian, frequentist, and other methodologies of statistics have tended to focus on conceptual justifications, sociological arguments, or mathematical proofs of their long run properties. Both Bayesian statistics and frequentist (“classical”) statistics have strong cases on these grounds. In this article, we instead approach the debates in the “Statistics Wars” from a largely unexplored angle: simulations of different methodologies’ performance in the short to medium run. We used Big Data methods to conduct a large number of simulations using a straightforward decision problem based around tossing a coin with unknown bias and then placing bets. In this simulation, we programmed four players, inspired by Bayesian statistics, frequentist statistics, Jon Williamson’s version of Objective Bayesianism, and a player who simply extrapolates from observed frequencies to general frequencies. The last player served a benchmark function: any worthwhile statistical methodology should at least match the performance of simplistic induction. We focused on the performance of these methodologies in guiding the players towards good decisions. Unlike an earlier simulation study of this type, we found no systematic difference in performance between the Bayesian and frequentist players, provided the Bayesian used a flat prior and the frequentist used a low confidence level. The Williamsonian player was also able to perform well given a low confidence level. However, the frequentist and Williamsonian players performed poorly with high confidence levels, while the Bayesian was surprisingly harmed by biased priors. Our study indicates that all three methodologies should be taken seriously by philosophers and practitioners of statistics.

Fast Methods for Drug Approval: Research Perspectives for Pandemic Preparedness

“…Unfortunately, Bayesian evidence synthesis requires the specification of a large number of relevant probabilities, which cannot be defined in an objectively correct way and must be fixed subjectively [116]. The debate on the seriousness of this subjectivity has been raging for many decades [117][118][119][120]. Let it suffice to say here that an end of this debate is not in sight.…”

Introduction to recent issues in philosophy of statistics: evidence, testing, and applications

“…We present a cross-section of these subjects, written by scholars from a variety of fields in order to explore issues in philosophy of statistics from different perspectives.The articles in this collection can be divided into roughly two categories. The first group contain articles by Mayo and Hand (2022), Radzvilas et al (2021), Rubin (2021), and Spanos (2021, and are concerned mainly with foundational issues in philosophy of statistics. In particular, the authors address questions on the procedure of statistical significance testing and its accompanying concepts of p-values and significance thresholds, Bayesian versus frequentist ("classical") statistics, and Ber-…”

E‐synthesis for carcinogenicity assessments: A case study of processed meat

“…In this study, we consider its applicability as a tool for food carcinogenicity assessments exemplified by an application to the disputed causal relationship between processed meat and cancer. E-Synthesis can address the above mentioned challenges for aggregating bodies of evidence on food carcinogenicity, as it (1) seamlessly aggregates evidence from different methodologies (randomized studies, observational studies and mechanistic evidence), (2) it outputs a Bayesian probability 26 that a food causes an adverse health effect in a population of food consumers that is ready-to-use for decision making and (3) it makes a number of judgements explicit and open to inspection and criticism that may remain opaque otherwise.…”