Redefine statistical significance

Benjamin, Daniel J.; Johannesson, Magnus; Nosek, Brian A.; Wagenmakers, Eric‐Jan; Berk, Richard A.; Bollen, Kenneth A.; Brembs, Björn; Brown, Lawrence D.; Camerer, Colin F.; Cesarini, David; Chambers, Chris; Clyde, Merlise Aycock; Cook, Tom; Boeck, Paul De; Dienes, Zoltán; Dreber, Anna; Easwaran, Kenny; Efferson, Charles; Fehr, Ernst; Fidler, Fiona; Field, Andy P.; Forster, Malcolm E.; George, Ed; Gonzalez, Richard; Goodman, Steve; Green, Ed; Green, Don; Greenwald, Anthony G.; Hadfield, Jarrod D.; Hedges, Larry V.; Held, Leonhard; Ho, Teck H.; Hoijtink, Herbert; Jones, James Holland; Hruschka, Daniel J.; Imai, Kosuke; Imbens, Guido W.; Ioannidis, John P. A.; Jeon, Minjeong; Kirchler, Michael; Laibson, David; List, John A.; Little, Roderick J. A.; Lupia, Arthur; Machery, Édouard; Maxwell, Scott E.; McCarthy, Michael; Morgan, Stephen L.; Munafò, Marcus R; Nakagawa, Shinichi; Nyhan, Brendan; Parker, Timothy; Pericchi, Luis R.; Perugini, Marco; Rouder, Jeffrey N.; Rousseau, Judith; Savalei, Victoria; Schönbrodt, Felix D.; Sellke, Thomas; Sinclair, Betsy; Tingley, Dustin; Zandt, Trisha Van; Vazire, Simine; Watts, Duncan J.; Winship, Christopher; Wolpert, Robert L.; Xie, Yuan; YOUNG, CRISTOBAL; Zinman, Jonathan; Johnson, Valen E.

doi:10.31234/osf.io/mky9j

Cited by 85 publications

(148 citation statements)

References 6 publications

(6 reference statements)

Supporting

Mentioning

141

Contrasting

Unclassified

Order By: Relevance

“…For example, with variable thresholds, many old problems with significance testing remain unsolved, such as the problems of regression to the mean of p-values, inflation of effect sizes (the "winner's curse", see below), selective reporting and publication bias, and the general disadvantage of forcing decisions too quickly rather than considering cumulative evidence across experiments. In view of all the uncertainty surrounding statistical inference (Greenland 2017(Greenland , 2018Amrhein et al, 2018), we strongly doubt that we could successfully "control" error rates if only we would justify our alpha level and other decisions in advance of a study, as Lakens et al (2018) seem to suggest in their comment to Benjamin et al (2018). Nonetheless, Lakens et al (2018) conclude that "the term 'statistically significant' should no longer be used."…”

Section: Error Rates and Variable Alpha Levelsmentioning

confidence: 99%

“…Sometimes, it is recommended that the alpha level be reduced to a more conservative value, to lower the Type I error rate. For example, Melton (1962) Benjamin et al (2018) did argue that using a .005 cutoff would fix much of what is wrong with significance testing. Unfortunately, as we will demonstrate, the problems with significance tests cannot be importantly mitigated merely by having a more conservative rejection criterion, and some problems are exacerbated by adopting a more conservative criterion.…”

Section: Introductionmentioning

confidence: 99%

“…We commence with some claims on the part of Benjamin et al (2018). For example, they wrote "…changing the P value threshold is simple, aligns with the training undertaken by many researchers, and might quickly achieve broad acceptance."…”

Section: Introductionmentioning

confidence: 99%

“…Finally, we acknowledge that Benjamin et al (2018) provided a service for science by further stimulating debate about significance testing. But there are important issues Benjamin et al (2018) seem not to have considered, discussed in the following sections.…”

Section: Introductionmentioning

confidence: 99%

“…But there are important issues Benjamin et al (2018) seem not to have considered, discussed in the following sections.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Manipulating the alpha level cannot cure significance testing

Trafimow

Amrhein

Areshenkoff

et al. 2018

Preprint

View full text Add to dashboard Cite

We argue that making accept/reject decisions on scientific hypotheses, including a recent call for changing the canonical alpha level from p= .05 to .005, is deleterious for the finding of new discoveries and the progress of science. Given that blanket and variable alpha levels both are problematic, it is sensible to dispense with significance testing altogether. There are alternatives that address study design and sample size much more directly than significance testing does; but none of the statistical tools should be taken as the new magic method giving clear-cut mechanical answers. Inference should not be based on single studies at all, but on cumulative evidence from multiple independent studies. When evaluating the strength of the evidence, we should consider, for example, auxiliary assumptions, the strength of the experimental design, and implications for applications. To boil all this down to a binary decision based on a p-value threshold of .05, .01, .005, or anything else, is not acceptable.

show abstract

Section: Error Rates and Variable Alpha Levelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…But there are important issues Benjamin et al (2018) seem not to have considered, discussed in the following sections.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Manipulating the alpha level cannot cure significance testing

Trafimow

Amrhein

Areshenkoff

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

What Is Better Than Coulomb Failure Stress? A Ranking of Scalar Static Stress Triggering Mechanisms from 10⁵ Mainshock‐Aftershock Pairs

Meade

DeVries

Faller

et al. 2017

Geophysical Research Letters

View full text Add to dashboard Cite

Aftershocks may be triggered by the stresses generated by preceding mainshocks. The temporal frequency and maximum size of aftershocks are well described by the empirical Omori and Bath laws, but spatial patterns are more difficult to forecast. Coulomb failure stress is perhaps the most common criterion invoked to explain spatial distributions of aftershocks. Here we consider the spatial relationship between patterns of aftershocks and a comprehensive list of 38 static elastic scalar metrics of stress (including stress tensor invariants, maximum shear stress, and Coulomb failure stress) from 213 coseismic slip distributions worldwide. The rates of true‐positive and false‐positive classification of regions with and without aftershocks are assessed with receiver operating characteristic analysis. We infer that the stress metrics that are most consistent with observed aftershock locations are maximum shear stress and the magnitude of the second and third invariants of the stress tensor. These metrics are significantly better than random assignment at a significance level of 0.005 in over 80% of the slip distributions. In contrast, the widely used Coulomb failure stress criterion is distinguishable from random assignment in only 51–64% of the slip distributions. These results suggest that a number of alternative scalar metrics are better predictors of aftershock locations than classic Coulomb failure stress change.

show abstract

Effects of serum from breast cancer surgery patients receiving perioperative dexmedetomidine on breast cancer cell malignancy: A prospective randomized controlled trial

Liu

Sun

et al. 2019

Cancer Medicine

View full text Add to dashboard Cite

Adrenergic receptors (ARs) have gained attention for their involvement in breast cancer (BC) progression. Dexmedetomidine, a selective α2‐AR agonist, has been reported to increase the malignancy of BC cells in vitro or stimulate tumor growth in mice. However, clinical evidence is lacking. Clinical research in this area is important as dexmedetomidine is widely used in BC surgery patients. Here we allocated 24 women with primary BC to the dexmedetomidine group (who received a total dose of 2 μg kg−1 dexmedetomidine perioperatively) or to the control group (who received the same volume of normal saline). Venous blood was obtained from all patients immediately upon entering the operating room and 24 hours postoperatively. Serum was then exposed to MCF‐7 cells at a concentration of 10% for 24 hours. Cell proliferation, migration, and invasion were analyzed using EdU, Transwell, and Matrigel methods, respectively. We found that postoperative serum from those who received dexmedetomidine was associated with significantly increased cell proliferation, migration, and invasion compared with preoperative serum when used to culture MCF‐7 cells. The mean percentage change from post to preoperative values in these cell functions was significantly larger in the dexmedetomidine group than in the control group (proliferation, 30.44% vs 8.45%, P = .0024; migration, 15.90% vs 3.25%, P = .0015; invasion, 8.17% vs 2.13%, P = .04). In conclusion, these findings suggest that in patients undergoing surgery for primary BC, perioperative administration of dexmedetomidine might influence the serum milieu in a way that favors the malignancy of MCF‐7 cells. Clinical trial registration: NCT03108937.

show abstract

Redefine statistical significance

Abstract: "We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."

Cited by 85 publications

References 6 publications

Manipulating the alpha level cannot cure significance testing

Manipulating the alpha level cannot cure significance testing

What Is Better Than Coulomb Failure Stress? A Ranking of Scalar Static Stress Triggering Mechanisms from 10⁵ Mainshock‐Aftershock Pairs

Effects of serum from breast cancer surgery patients receiving perioperative dexmedetomidine on breast cancer cell malignancy: A prospective randomized controlled trial

Contact Info

Product

Resources

About

Redefine statistical significance

Abstract: "We propose to change the default P-value threshold forstatistical significance for claims of new discoveries from 0.05 to 0.005."

Cited by 85 publications

References 6 publications

Manipulating the alpha level cannot cure significance testing

Manipulating the alpha level cannot cure significance testing

What Is Better Than Coulomb Failure Stress? A Ranking of Scalar Static Stress Triggering Mechanisms from 105 Mainshock‐Aftershock Pairs

Effects of serum from breast cancer surgery patients receiving perioperative dexmedetomidine on breast cancer cell malignancy: A prospective randomized controlled trial

Contact Info

Product

Resources

About

What Is Better Than Coulomb Failure Stress? A Ranking of Scalar Static Stress Triggering Mechanisms from 10⁵ Mainshock‐Aftershock Pairs