Can a registered trial be reported as a one-group, pretest-posttest study with no explanation? A critique of Williams et al. (2021)

Gorman, Dennis M.

doi:10.1186/s40352-021-00165-3

Cited by 2 publications

(1 citation statement)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Benjamin et al [17] acknowledge that an investigator can still engage in such analytic practices while using a P<0.005 threshold but contend that the likelihood of a "statistically significant" chance finding emerging from these analyses is lower than with a 0.05 threshold. Moreover, it is likely that the longer the data dredging exercise continues, the more easily identifiable it will be to those reading a publication in which such results are reported; that is, investigators will be forced into conducting more extreme analyses as the search for a statistically significant result continues (e.g., removing study groups or assessment points from the analysis, even when the study is registered) [e.g., 20,21].…”

Section: Introductionmentioning

confidence: 99%

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Fitzpatrick,

Gorman,

Trombatore

2024

PLoS ONE

Self Cite

View full text Add to dashboard Cite

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote “statistical significance” and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

show abstract

Section: Introductionmentioning

confidence: 99%

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Fitzpatrick,

Gorman,

Trombatore

2024

PLoS ONE

Self Cite

View full text Add to dashboard Cite

show abstract

Commentary: Effectiveness of a hybrid digital substance abuse prevention approach combining e-learning and in-person class sessions

Gorman

2023

Front. Digit. Health

Self Cite

View full text Add to dashboard Cite

This commentary contends that the results pertaining to substance use frequency reported in Griffin, Williams, Botvin, Sousa, and Botvin (2022) "Effectiveness of a hybrid digital substance abuse prevention approach combining e-Learning and in-person class sessions", Frontiers in Digital Health are of no practical significance and are likely chance findings that emerge from the very liberal approach to statistical significance testing employed in the data analysis.Griffin et al. conclude that their study "showed significant reductions in substance use" and that their results "demonstrate that substance abuse prevention programs conducted during middle school using hybrid e-learning modules plus a limited number of classroom sessions can produce meaningful reductions in substance use and improve important life skills" [(1), pp. 13−14]. These statements are based entirely on the statistically significant differences between the study conditions and take no account of the practical significance of the results.Griffin et al. present two statistical models estimating the effects of the intervention, a GLM model and a MIXED model. The latter is the more rigorous as it accounts for clustering of students within the schools randomized to the two study conditions. Figure 1 presents the substance use outcomes for which a statistically significant difference was reported at posttest between the intervention and control groups using each model. It shows that the mean substance use of both groups on all outcomes occurs at the very low end of the 9-point Likert Scale, specifically between 1 ("Never") and 2 ("A few times but NOT in the past year"). Given the age of the subjects, this is unsurprising. But, most noticeably, the differences between the two groups are very small on all outcomes: 0.06−0.09 for the six GLM model results and 0.09−0.14 for the three MIXED model results. The best TYPE General Commentary

show abstract

Can a registered trial be reported as a one-group, pretest-posttest study with no explanation? A critique of Williams et al. (2021)

Cited by 2 publications

References 4 publications

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Commentary: Effectiveness of a hybrid digital substance abuse prevention approach combining e-learning and in-person class sessions

Contact Info

Product

Resources

About