An Ethical Approach to Peeking at Data

Sagarin, Brad J.; Ambler, James K.; Lee, Ellen M.

doi:10.1177/1745691614528214

Cited by 107 publications

(114 citation statements)

References 27 publications

(56 reference statements)

Supporting

Mentioning

105

Contrasting

Unclassified

Order By: Relevance

“…Thus, current best practices were used to quantify this risk (Sagarin, Ambler & Lee, 2014). In the SOM, the P augmented statistic for each dependent variable is reported.…”

Section: Methodsmentioning

confidence: 99%

Is spending money on others good for your heart?

et al. 2016

View full text Add to dashboard Cite

show abstract

“…Thus, current best practices were used to quantify this risk (Sagarin, Ambler & Lee, 2014). In the SOM, the P augmented statistic for each dependent variable is reported.…”

Section: Methodsmentioning

confidence: 99%

Is spending money on others good for your heart?

et al. 2016

View full text Add to dashboard Cite

show abstract

“…To answer this sort of question, the analyst must take into account the exact stopping and testing intentions. Consequently, there is a proliferation of methods for computing correct p values and confidence intervals for many different stopping and testing situations (e.g., Sagarin, Ambler, & Lee, 2014). It is important to understand that any such procedure does not yield the credibility of parameter values (as in Eq.…”

Section: Needless To Say P(rain|noon) Does Not Equal P(noon|rain) Amentioning

confidence: 99%

The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective

2017

View full text Add to dashboard Cite

In the practice of data analysis, there is a conceptual distinction between hypothesis testing, on the one hand, and estimation with quantified uncertainty on the other. Among frequentists in psychology, a shift of emphasis from hypothesis testing to estimation has been dubbed "the New Statistics" (Cumming, 2014). A second conceptual distinction is between frequentist methods and Bayesian methods. Our main goal in this article is to explain how Bayesian methods achieve the goals of the New Statistics better than frequentist methods. The article reviews frequentist and Bayesian approaches to hypothesis testing and to estimation with confidence or credible intervals. The article also describes Bayesian approaches to meta-analysis, randomized controlled trials, and power analysis.Keywords Null hypothesis significance testing · Bayesian inference · Bayes factor · Confidence interval · Credible interval · Highest density interval · Region of practical equivalence · Meta-analysis · Power analysis · Effect size · Randomized controlled trial · Equivalence testingThe New Statistics emphasizes a shift of emphasis away from null hypothesis significance testing (NHST) to "estimation based on effect sizes, confidence intervals, and metaanalysis" (Cumming, 2014, p. 7). There are many reasons John K. Kruschke johnkruschke@gmail.com 1 Indiana University, Bloomington, USA to eschew NHST, with its seductive lapse to black-and-white thinking about the presence or absence of effects. There are also many reasons to promote instead a cumulative science that incrementally improves estimates of magnitudes and uncertainty. These reasons were recently highlighted in a prominent statement from the American Statistical Association (ASA; Wasserstein & Lazar, 2016) that will be summarized later in this article. Recent decades have also seen repeated calls to shift emphasis away from frequentist methods to Bayesian analysis (e.g., Lindley, 1975).In this article, we review both of these recommended shifts of emphasis in the practice of data analysis, and we promote their convergence in Bayesian methods for estimation. The goals of the New Statistics are better achieved by Bayesian methods than by frequentist methods. In that sense, we recommend a Bayesian New Statistics. Within the domain of Bayesian methods, we have a more nuanced emphasis. Bayesian methods provide a coherent framework for hypothesis testing, so when null hypothesis testing is the crux of the research then Bayesian null hypothesis testing should be carefully used. But we also believe that typical analyses should not routinely stop with hypothesis testing alone. In that sense, we recommend a New Bayesian Statistics, that is, Bayesian analyses that also consider estimates of magnitudes and uncertainty, along with meta-analyses.This article begins with an extensive description of frequentist and Bayesian approaches to null hypothesis testing and estimation with confidence or credible intervals. Subsequently, the article explains Bayesian approaches to metaanalysis, rando...

show abstract

“…Type-1 error inflation resulting from multiple looks, we also report p augmented for the focal tests (Sagarin, Ambler, & Lee, 2014; SOM for details).…”

Section: Resultsmentioning

confidence: 99%