Reevaluating excess success in psychological science

Abstract

Francis (Psychonomic Bulletin Review, 21, 1180-1187, 2014 recently claimed that 82 % of articles with four or more experiments published in Psychological Science between 2009 and 2012 cannot be trusted. We critique Francis' analysis and point out the dependence of his approach on including the appropriate experiments and significance tests.

Confirming the appearance of excess success: Reply to van Boxtel and Koch (2016)

“…To satisfy this requirement, only a single test from Experiment 1 could be part of the p-curve analysis, even though the theoretical claim depended on multiple successful tests. The same problems apply to the p-curve analyses for other papers reported in Table S1 of van Boxtel and Koch (2016).…”

“…Setting aside the calculation errors of success probabilities in van Boxtel and Koch (2016), their primary disagreement with the analysis in Francis (2014) concerns which tests were relevant to the theoretical claims in van Boxtel and Koch (2012). Readers will have to check the text of van Boxtel and Koch (2012) to verify that the description of the tests in Table 1 properly represents their original reasoning.…”

“…Van Boxtel and Koch (2016) refer to Ioannidis (2013) when claiming that the TES analysis is valid only when all the tests investigate the same hypothesis, but the claim is not true, and Ioannidis (2013) makes no such statement. Van Boxtel and Koch (2016) also claimed that the analyses in Francis (2014) used Ba single overall average effect size to estimate power.^Contrary to their claim, the supplemental material of Francis (2014) explains how the success probabilities were computed with simulated experiments that used the sample statistics as estimated population parameters. The supplemental material of Francis (2014) includes source code for these simulations, so readers can examine the details of the analysis for themselves.…”

“…Van Boxtel and Koch (2016) next suggested that the estimated joint success probability Bpoorly reflects the strength of the findings,^apparently because their calculated p = .56 does not indicate a significant effect and is much smaller than the average power (0.93). In addition to mistakenly interpreting the estimated joint success probability as a p value, van Boxtel and Koch (2016) seem to have fallen victim to the Bconjunction fallacy,^which violates the fundamental property of probability that a joint set of outcomes must always be no more probable than any subset of those outcomes (e.g., Tversky & Kahneman, 1983).…”

“…Van Boxtel and Koch (2016) subsequently calculated another estimate of success probability by averaging the success probabilities of the different tests and then using the properties of the binomial distribution to compute the probability of observing success in all eight tests. This averaging process is inappropriate, as can be seen by an example.…”