Simultaneous and selective inference: Current successes and future challenges

Benjamini, Yoav

doi:10.1002/bimj.200900299

Cited by 159 publications

(121 citation statements)

References 31 publications

Supporting

Mentioning

117

Contrasting

Unclassified

Order By: Relevance

“…So Q can be considered a random variable. However, as Q cannot be controlled directly FDR is defined as the expected value of the proportion of false positive errors: FDR = E[FP/R|R > 0] · pr(R > 0), a variable which can be controlled (see Benjamini and Hochberg, 1995;Curran-Everett, 2000;Nichols and Hayasaka, 2003;Bennett et al, 2009;Benjamini, 2010;Goeman and Solari, 2014). Some FDR estimation procedures can also factor in dependency between tests (Benjamini and Yekutieli, 2001).…”

Section: Family-wise Error Rate (Fwer) and Fdr Correction In Nhstmentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

Null hypothesis significance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions. Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative findings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out.

show abstract

Section: Family-wise Error Rate (Fwer) and Fdr Correction In Nhstmentioning

confidence: 99%

When null hypothesis significance testing is unsuitable for research: a reassessment

Szűcs

Ioannidis

2016

Preprint

107

View full text Add to dashboard Cite

show abstract

“…Correcting the diagnosticity of P values for the number of tests that were actually conducted is relatively straightforward (33,34), although inconsistently--even rarely--applied in practice (35,36). However, counting the literal performance of statistical tests is not sufficient to account for how observing the data can influence the selection of tests to conduct.…”

Section: Standard Tools Of Statistical Inference Assume Predictionmentioning

confidence: 99%

“…If one conducts 10 analyses and the narrative implications of the paper focus on just two of them, inferential error can increase in how the paper is applied and cited. Essentially this is a circumstance of failing to correct for multiple comparisons (35). This can be corrected statistically by applying alpha corrections like Bonferroni (56) such that narrative focus on just positive results is not associated with inflating likelihood of false positives.…”

Section: Preregistration In Practicementioning

confidence: 99%

The preregistration revolution

Nosek

Ebersole

DeHaven

et al. 2018

Proc. Natl. Acad. Sci. U.S.A.

1,523

1,267

View full text Add to dashboard Cite

Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations. This distinction between postdiction and prediction is appreciated conceptually but is not respected in practice. Mistaking generation of postdictions with testing of predictions reduces the credibility of research findings. However, ordinary biases in human reasoning, such as hindsight bias, make it hard to avoid this mistake. An effective solution is to define the research questions and analysis plan before observing the research outcomes-a process called preregistration. Preregistration distinguishes analyses and outcomes that result from predictions from those that result from postdictions. A variety of practical strategies are available to make the best possible use of preregistration in circumstances that fall short of the ideal application, such as when the data are preexisting. Services are now available for preregistration across all disciplines, facilitating a rapid increase in the practice. Widespread adoption of preregistration will increase distinctiveness between hypothesis generation and hypothesis testing and will improve the credibility of research findings.methodology | open science | confirmatory analysis | exploratory analysis | preregistration P rogress in science is marked by reducing uncertainty about nature. Scientists generate models that may explain prior observations and predict future observations. Those models are approximations and simplifications of reality. Models are iteratively improved and replaced by reducing the amount of prediction error. As prediction error decreases, certainty about what will occur in the future increases. This view of research progress is captured by George Box's aphorism: "All models are wrong but some are useful" (1, 2).Scientists improve models by generating hypotheses based on existing observations and testing those hypotheses by obtaining new observations. These distinct modes of research are discussed by philosophers and methodologists as hypothesis-generating versus hypothesis-testing, the context of discovery versus the context of justification, data-independent versus data-contingent analysis, and exploratory versus confirmatory research (e.g., refs. 3-6). We use the more general terms--postdiction and prediction--to capture this important distinction.A common thread among epistemologies of science is that postdiction is characterized by the use of data to generate hypotheses about why something occurred, and prediction is characterized by the acquisition of data to test ideas about what will occur. In prediction, data are used to confront the possibility that the prediction is wrong. In postdiction, the data are already known and the postdiction is generated to explain why they occurred.Testing predictions is vital for establishing diagnostic evidence for explanatory claims. Testing predictions assesses the uncertainty of scientific models by observing how well the predictions account for new data. Generating postd...

show abstract

“…Besides point estimation, hypothesis testing in experiments with multiple treatments also suffers from an issue called multiple comparison (Benjamini 2010). Framework of hypothesis testing only guarantees Type I error (False Positive Rate) be controlled under the significant size, which is usually set to 5%, when there is only one test.…”

Section: Multiple Treatments In A/b Testmentioning

confidence: 99%

Statistical inference in two-stage online controlled experiments with treatment selection and validation

Deng

Guo

2014

Proceedings of the 23rd International Conference on World Wide Web

View full text Add to dashboard Cite

Online controlled experiments, also called A/B testing, have been established as the mantra for data-driven decision making in many web-facing companies. A/B Testing support decision making by directly comparing two variants at a time. It can be used for comparison between (1) two candidate treatments and (2) a candidate treatment and an established control. In practice, one typically runs an experiment with multiple treatments together with a control to make decision for both purposes simultaneously. This is known to have two issues. First, having multiple treatments increases false positives due to multiple comparison. Second, the selection process causes an upward bias in estimated effect size of the best observed treatment. To overcome these two issues, a two stage process is recommended, in which we select the best treatment from the first screening stage and then run the same experiment with only the selected best treatment and the control in the validation stage. Traditional application of this two-stage design often focus only on results from the second stage. In this paper, we propose a general methodology for combining the first screening stage data together with validation stage data for more sensitive hypothesis testing and more accurate point estimation of the treatment effect. Our method is widely applicable to existing online controlled experimentation systems.

show abstract

Simultaneous and selective inference: Current successes and future challenges

Cited by 159 publications

References 31 publications

When null hypothesis significance testing is unsuitable for research: a reassessment

When null hypothesis significance testing is unsuitable for research: a reassessment

The preregistration revolution

Statistical inference in two-stage online controlled experiments with treatment selection and validation

Contact Info

Product

Resources

About