Killeen's probability of replication and predictive probabilities: How to compute, use, and interpret them.

Abstract

P. R. Killeen's (2005a) probability of replication (prep) of an experimental result is the fiducial Bayesian predictive probability of finding a same-sign effect in a replication of an experiment. prep is now routinely reported in Psychological Science and has also begun to appear in other journals. However, there is little concrete, practical guidance for use of prep, and the procedure has not received the scrutiny that it deserves. Furthermore, only a solution that assumes a known variance has been implemented. A practical problem with prep is identified: In many articles, prep appears to be incorrectly computed, due to the confusion between 1-tailed and 2-tailed p values. Experimental findings reveal the risk of misinterpreting prep as the predictive probability of finding a same-sign and significant effect in a replication (p srep). Conceptual and practical guidelines are given to avoid these pitfalls. They include an extension to the case of unknown variance. Moreover, other uses of fiducial Bayesian predictive probabilities for analyzing, designing ("how many subjects?"), and monitoring ("when to stop?") experiments are presented. Concluding remarks emphasize the role of predictive procedures in statistical methodology.

Nonparametric Predictive Inference for Reproducibility of Basic Nonparametric Tests

“…The paper is accompanied by a discussion that makes clear that also this concept is somewhat confusing, but the general ideas about RP in Killeen's paper are not too distant from those presented in this article, namely, the predictive nature of RP, which we explicitly use, and the informal way of considering RP as the "real power" of a test, with "power" interpreted in its everyday, so nonstatistical, meaning, which we also support. Lecoutre et al (2010) discuss Killeen's approach further, referring to it as "fiducial Bayesian predictive probability," and mentioning that it is now increasingly popular. They discuss some problems in its computation, resulting again from some apparent confusions.…”

An IRT analysis of motive questionnaires: The Unified Motive Scales

“…If an exact replication is carried out (i.e., using exactly the same measures and the same sample size, but a new random sample), the probability of finding a significant correlation in the same direction is 71% (p srep ; Killeen, 2005;Lecoutre, Lecoutre, & Poitevineau, 2010). If the replication is done with a (seemingly) equivalent predictor measure with a convergent correlation of .57, p srep drops to 38%.…”

Replacing P-values with frequentist posterior probabilities of replication—When possible parameter values must have uniform marginal prior probabilities

“…Kileen suggested calculating a probability of replicating a difference between an experimental result and control in the same direction when a study is repeated, based on the original P value alone [8]. For example, if the one-sided P value was 0.025 then the P-rep would be about (1+(0.025/(1−0.025)) 2/3 )−1 = 0.92 [8, 9]. However, this result, which is based on a number of disputed assumptions [9], would be inconsistent with the probability of 0.975 of the null hypothesis or something more extreme by assuming a uniform or weak distribution for all possible hypothetical outcomes and applying a Bayesian calculation [3, 4, 5, 6].…”

“…For example, if the one-sided P value was 0.025 then the P-rep would be about (1+(0.025/(1−0.025)) 2/3 )−1 = 0.92 [8, 9]. However, this result, which is based on a number of disputed assumptions [9], would be inconsistent with the probability of 0.975 of the null hypothesis or something more extreme by assuming a uniform or weak distribution for all possible hypothetical outcomes and applying a Bayesian calculation [3, 4, 5, 6]. Also there is no provision in Kileen’s P-rep approach for incorporating the probable effect of other data or methodological irregularities into the probability of replicating a result (e.g.…”