The Extent and Consequences of P-Hacking in Science

Head, Megan L.; Holman, Luke; Lanfear, Robert; Kahn, Andrew T.; Jennions, Michael D.

doi:10.1371/journal.pbio.1002106

Cited by 1,018 publications

(980 citation statements)

References 60 publications

Supporting

Mentioning

936

Contrasting

Unclassified

Order By: Relevance

“…One example is by Head, Holman, Lanfear, Kahn, and Jennions (2015), who p-curved all p values published in Open Access journals. The article asks an arguably meaningless question-"What is the evidential value of all tests, whether relevant or 3 They do also simulate less extreme versions (e.g., collecting only two measures and reporting the lowest p value of those two).…”

Section: Problem 1: P-curvers Not Following Directionsmentioning

confidence: 99%

Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Simonsohn

Simmons

Nelson

2015

Journal of Experimental Psychology: General

288

349

View full text Add to dashboard Cite

When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01 s) than high (.04 s) p values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (a) p-curvers selecting the wrong p values, (b) fake data, (c) honest errors, and (d) ambitiously p-hacked (beyond p Ͻ .05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.

show abstract

Section: Problem 1: P-curvers Not Following Directionsmentioning

confidence: 99%

Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Simonsohn

Simmons

Nelson

2015

Journal of Experimental Psychology: General

288

349

View full text Add to dashboard Cite

show abstract

“…It has been proposed recently that, if authors make post-hoc decisions based on inspection of the data to meet the significance criterion of alpha = .05, then results just around this cutoff will be over-represented compared to results that are further away from this cutoff (e.g., Simonsohn, Nelson, & Simmons, 2014;Head, Holman, Lanfear, Kahn, & Jennions, 2015). This behavior, called 'p-hacking', could be detected in a body of literature by using a technique called 'p-curving', which basically consists in plotting the distribution of significant p-values.…”

Section: Methodsmentioning

confidence: 99%

Can infants learn phonology in the lab? A meta-analytic answer

Cristià¹

2016

Preprint

View full text Add to dashboard Cite

Two of the key tasks facing the language-learning infant lie at the level of phonology: establishing which sounds are contrastive in the native inventory, and determining what their possible syllabic positions and permissible combinations (phonotactics) are. In 2002-2003, two theoretical proposals, one bearing on how infants can learn sounds (Maye, Werker, & Gerken, 2002) and the other on phonotactics (K. E. Chambers, Onishi, & Fisher, 2003), were put forward on the pages of Cognition, each supported by two laboratory experiments, wherein a group of infants was briefly exposed to a set of pseudo-words, and plausible phonological generalizations were tested in a subsequent phase. These two papers have received considerable attention from the general scientific community, and inspired a flurry of follow-up work. In the context of questions regarding the replicability of psychological science, the present work uses a meta-analytic approach to appraise extant empirical evidence for infant phonological learning in the laboratory. It is found that neither seminal finding (on learning sounds and learning phonotactics) holds up when close methodological replications are integrated, although less close methodological replications do provide some evidence in favor of the sound learning strand of work. Implications for authors and readers of this literature are drawn out. It would be desirable that additional mechanisms for phonological learning be explored, and that future infant laboratory work should employ paradigms that rely on constrained and unambiguous links between experimental exposure and measured infant behavior.

show abstract

“…Second, P-values can be easily 'hacked' (Head, Holman, Lanfear, Kahn, & Jennions, 2015;Simmons, Nelson, & Simonsohn, 2011). That is, researchers can engage in questionable research practices (QRPs; John, Loewenstein, & Prelec, 2012) to obtain 'significant' P-values that are less than .05, which is accepted widely by convention (Nelson, Simmons, & Simonsohn, 2018).…”

Section: Running Head: Utilizing Bayesian Statisticsmentioning

confidence: 99%

Why do we need to employ Bayesian statistics and how can we employ it in studies of moral education?: With practical guidelines to use JASP for educators and researchers

Han

Park

Thoma

2018

Journal of Moral Education

View full text Add to dashboard Cite

In this paper, we discuss the benefits of and how to utilize Bayesian statistics in studies of moral education. To demonstrate concrete examples of the applications of Bayesian statistics to studies of moral education, we reanalyzed two datasets previously collected: one small dataset collected from a moral educational intervention experiment, and one big dataset from a largescale Defining Issues Test-2 survey. Results suggest that Bayesian analysis of datasets collected from moral educational studies can provide additional useful statistical information, particularly that associated with the strength of evidence supporting alternative hypotheses, which has not been provided by the classical frequentist approach focusing on P-values. Finally, we introduce several practical guidelines pertaining to how to utilize Bayesian statistics, including the utilization of newly developed free statistical software, Jeffrey's Amazing Statistics Program (JASP), and thresholding based on Bayes Factors, to scholars in the field of moral education.

show abstract

The Extent and Consequences of P-Hacking in Science

Cited by 1,018 publications

References 60 publications

Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Can infants learn phonology in the lab? A meta-analytic answer

Why do we need to employ Bayesian statistics and how can we employ it in studies of moral education?: With practical guidelines to use JASP for educators and researchers

Contact Info

Product

Resources

About