Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015).

Abstract

When studies examine true effects, they generate right-skewed p-curves, distributions of statistically significant results with more low (.01 s) than high (.04 s) p values. What else can cause a right-skewed p-curve? First, we consider the possibility that researchers report only the smallest significant p value (as conjectured by Ulrich & Miller, 2015), concluding that it is a very uncommon problem. We then consider more common problems, including (a) p-curvers selecting the wrong p values, (b) fake data, (c) honest errors, and (d) ambitiously p-hacked (beyond p Ͻ .05) results. We evaluate the impact of these common problems on the validity of p-curve analysis, and provide practical solutions that substantially increase its robustness.

A meta-analysis of the survival-processing advantage in memory

“…If a field contains evidential value, the p-value distribution will appear rightskewed (Simonsohn, Nelson, & Simmons, 2014). Along with testing for evidential value, pcurve analysis also provides statistical power estimates after correcting for selective reporting (Simonsohn, Simmons, & Nelson, 2015). p-curve analyses were performed using the online application at p-curve.com (Simonsohn et al, 2014 …”

A meta-analysis of the P3 amplitude in tasks requiring deception in legal and social contexts

“…From a methodological perspective, it is of interest to compare different descriptive and quantitative techniques (e.g., trim and fill, funnel plot, fail-safe number, pcurve analysis) to control for publication bias (Duval & Tweedie, 2000;Heene, 2010;Orwin, 1983;Simonsohn, Simmons, & Nelson, 2015). To correct measurement errors of the P3 amplitude more closely in conjunction with the P3 quantification method, future studies should take reliability of difference scores into account when peak-to-peak P3 quantification is applied (e.g., Overall & Woodward, 1975).…”

Do infants have a sense of numerosity? A p-curve analysis of infant numerosity discrimination studies

“…The half p-curve is more robust against p-hacking because it assesses the distribution of p-values < .025. In doing so, the half p-curve is less likely to mistake phacking for evidential value (Simonsohn et al, 2015). Lakens (2014) nicely summarizes the value of p-curve analysis in controlling for publication bias by saying, "traditional meta-analyses are one approach, but suffer from publication bias.…”

“…The half p-curve assesses the p-values that fall < .025, whereas the full p-curve assesses the p-values that fall across the entire p < .05 spectrum. The half p-curve provides a more robust analysis against p-hacking (Simonsohn, Simmons, & Nelson, 2015). The half p-curve is more robust against p-hacking because it assesses the distribution of p-values < .025.…”