Principles of Inference and Their Consequences

Mayo, Deborah G.; Kruse, Michael

doi:10.1007/978-94-017-1586-7_16

Cited by 46 publications

(22 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Optional stopping, and particularly preferential stopping, can be used to further influence these frequencies. Though previous work has noted that the universal bound does not apply to composite hypotheses-e.g., in statistics (Mayo & Kruse, 2001;Royall, 2000, Rejoinder)-and the initial favoring of the null hypothesis in Bayes factors has been noted before in psychology and statistics (Atkinson, 1978;Matthews, 2011;Rouder et al, 2009), the work presented here provides a more thorough and general investigation of these findings, showing explicitly how they are related to violations of the assumptions of the universal bound. In particular, the present work demonstrates how large the influence of the choice of stopping rule can be.…”

Section: Discussionsupporting

confidence: 61%

“…As we will show, these assumptions are rarely met in psychology experiments. Indeed the limitations on the universal bound to tests between simple hypotheses has been noted before in the statistics literature (Mayo & Kruse, 2001;Royall, 2000, Rejoinder), but here we investigate the quantitative consequences of the limitations as well as go into greater depth as to where the universal bound remains a good guide. In the following two sections we explore the consequences of preferential stopping when the universal bound's assumptions are not satisfied.…”

Section: Optional Stopping and The Universal Boundmentioning

confidence: 79%

“…In particular, while the above quote about the bounds is correct, it applies only to very narrow situations (Mayo & Kruse, 2001;Royall, 2000, Rejoinder), situations that rarely occur in psychological research. As a consequence, the above bound cannot always be used as a guide for the frequentist implications of using Bayes factors in psychological research, and it is possible that the probabilities of generating convincing Bayes factors are far higher than the quote implies.…”

mentioning

confidence: 95%

See 2 more Smart Citations

The frequentist implications of optional stopping on Bayesian hypothesis tests

Sanborn

Hills

2013

Psychon Bull Rev

View full text Add to dashboard Cite

Null hypothesis significance testing (NHST) is the most commonly used statistical methodology in psychology. The probability of achieving a value as extreme or more extreme than the statistic obtained from the data is evaluated, and if it is low enough then the null hypothesis is rejected. However, because common experimental practice often clashes with the assumptions underlying NHST, these calculated probabilities are often incorrect. Most commonly, experimenters use tests that assume sample sizes are fixed in advance of data collection, but then use the data to determine when to stop; in the limit, experimenters can use data monitoring to guarantee the null hypothesis will be rejected. Bayesian hypothesis testing (BHT) provides a solution to these ills, because the stopping rule used is irrelevant to the calculation of a Bayes factor. In addition, there are strong mathematical guarantees on the frequentist properties of BHT that are comforting for researchers concerned that stopping rules could influence the Bayes factors produced. Here we show that these guaranteed bounds have limited scope and often do not apply in psychological research. Specifically, we quantitatively demonstrate the impact of optional stopping on the resulting Bayes factors in two common situations: 1) when the truth is a combination of the hypotheses, such as in a heterogeneous population, and 2) when a hypothesis is composite-taking multiple parameter values-such as the alternative hypothesis in a t-test. We found that for these situations that while the Bayesian interpretation remains correct regardless of the stopping rule used, the choice of stopping rule can, in some situations, greatly increase the chance of an experimenter finding evidence in the direction they desire. We suggest ways to control these frequentist implications of stopping rules on BHT.The standard statistical methods of psychological research, such as the ANOVA and the t-test, are examples of null hypothesis significance testing (NHST). These familiar frequentist methods compare a statistic calculated from the data, such as an F-test statistic or a t-test statistic, to a sampling distribution associated with the null hypothesis. If the probability of achieving a result

show abstract

Section: Discussionsupporting

confidence: 61%

Section: Optional Stopping and The Universal Boundmentioning

confidence: 79%

mentioning

confidence: 95%

See 1 more Smart Citation

The frequentist implications of optional stopping on Bayesian hypothesis tests

Sanborn

Hills

2013

Psychon Bull Rev

View full text Add to dashboard Cite

show abstract

“…The preceding arguments have dealt with the evidential, post-experimental 6 Note that this does not hold for improper priors where the integral over the probability density is not equal to 1, see Mayo and Kruse 2001. 7 A prima facie counterargument against Bayes factors consists in the "subjectivity" of the prior probabilities in B(H 1 , H 0 , ·). But priors can be reported separately and disentangled from the "impact of the evidence".…”

mentioning

confidence: 99%

Evidence and Experimental Design in Sequential Trials

Sprenger¹

2009

Philos. of Sci.

View full text Add to dashboard Cite

To what extent does the design of statistical experiments, in particular sequential trials, affect their interpretation? Should postexperimental decisions depend on the observed data alone, or should they account for the used stopping rule? Bayesians and frequentists are apparently deadlocked in their controversy over these questions. To resolve the deadlock, I suggest a three-part strategy that combines conceptual, methodological, and decision-theoretic arguments. This approach maintains the pre-experimental relevance of experimental design and stopping rules but vindicates their evidential, postexperimental irrelevance.

show abstract

“…Here, although the inferences (about the criminal, the deflection effect, the table length) were constructed to fit the data, they were deliberately constrained to reflect what is correct, at least approximately. It is the severity, stringency, or probativeness of the test-or lack of it-therefore that should determine if a double use of data is admissible-or so I have argued (Mayo [1991], [1996]; Mayo and Kruse [2001]; Mayo and Cox [2006]). Hitchcock and Sober ([2004]) question whether the severity criterion can perform this job: if it is interpreted so as to sanction the latter cases of reliable double-counting, they think, then it appears also to countenance the former cases of unreliable selection effects.…”

mentioning

confidence: 99%

How to Discount Double-Counting When It Counts: Some Clarifications

Mayo

2008

The British Journal for the Philosophy of Science

View full text Add to dashboard Cite

The issues of double-counting, use-constructing, and selection effects have long been the subject of debate in the philosophical as well as statistical literature. I have argued that it is the severity, stringency, or probativeness of the test-or lack of it-that should determine if a double-use of data is admissible. Hitchcock and Sober ([2004]) question whether this 'severity criterion' can perform its intended job. I argue that their criticisms stem from a flawed interpretation of the severity criterion. Taking their criticism as a springboard, I elucidate some of the central examples that have long been controversial, and clarify how the severity criterion is properly applied to them.

show abstract

Principles of Inference and Their Consequences

Cited by 46 publications

References 24 publications

The frequentist implications of optional stopping on Bayesian hypothesis tests

The frequentist implications of optional stopping on Bayesian hypothesis tests

Evidence and Experimental Design in Sequential Trials

How to Discount Double-Counting When It Counts: Some Clarifications

Contact Info

Product

Resources

About