Potential performance theory (PPT): Describing a methodology for analyzing task performance

Abstract

Potential performance theory (PPT) was developed by Trafimow and Rice (2008) as a general theory of task performance that specifies the relationships among actual performance, true performance, and randomness. Although Trafimow and Rice performed mathematical simulations of the theory's consequences given various starting points, and even performed an illustrative experiment, they did not fully explore the potential of PPT as a methodological tool. We believe that there are such implications and that they are important. Consequently, our main goals are to work out these implications in detail, perform an experiment to demonstrate them, and illustrate how the conclusions drawn from traditional methods are misleading in the absence of the proposed methods. Furthermore, we introduce new mathematical equations that further enhance the use of PPT. To aid in demonstrating this new methodology, we use an example paradigm, whereby participants perform a high-fidelity target-detection task; however, we want to be clear that our goal is to demonstrate the proposed PPT methodology, rather than to contribute extensively to the literature on this particular paradigm. A SUMMARY OF PPTPPT is based on the idea that a person's observed performance on a task is influenced by two variables: (1) the combination of the nonrandom factors that are relevant to the performance of the task, which PPT terms strategy, and (2) random factors. For those people whose strategy is better than randomness, it is easy to demonstrate that lack of consistency (randomness) decreases task performance (see Trafimow & Rice, 2008, Figure 1). PPT quantifies what the person's expected task performance would be if random factors were completely eliminated-that is, if the person were to execute a perfectly consistent pattern of behavior. In addition, PPT quantifies what the person's task performance would be under any level of consistency that the researcher desires to consider.PPT is based, in part, on classical true-score theory. According to classical true-score theory, observed performance is determined by two factors: a person's "true" score and random error (for reviews, see Allen & Yen, 1979;Cohen & Swerdlik, 1999;Crocker & Algina, 1986;Lord & Novick, 1968). Interestingly, although random error is, by definition, random, it can have systematic effects on correlation coefficients. As random error increases, correlations between variables decrease. Classical true-score theorists derived a "correction" formula that provides the expected value for the true correlation if the measures of the variables are perfectly reliable and have no random error. Specifically, the expected value for the true correlation coefficient is based on the observed correlation coefficient, adjusted for the reliability of the measures of the relevant variables. Equation 1 gives the correction formula, where R is the true correlation between variables X and Y, and r XX and r YY are the reliabilities of the measures of the two variables, respectively. R r r r XX YY(1) Because task ...

A measure of within-participant response consistency

“…This might be considered a within-participants version of test-retest reliability. Although this measure often works well (e.g., Trafimow & Rice, 2009), it cannot always be used. Sometimes the experimental design requires that there is only one block of trials, or the measure may be applied after the fact to a preexisting data set.…”

The Assumption of a Reliable Instrument and Other Pitfalls to Avoid When Considering the Reliability of Data

“…He similarly indicated that the observed correlation between two variables will only reach its theoretical maximum of 1 when (a) the reliability of the variables are perfect and (b) the correlation between the true score equivalents is equal to 1. (Readers may also consult Trafimow and Rice, 2009 for an interesting application of this correction formula to behavioral task performance via potential performance theory).…”

Using Potential Performance Theory to Analyze Systematic and Random Factors in Enumeration Tasks

“…According to PPT, one of the primary factors that contributes to general task performance is consistency (Rice, Trafimow, & Hunt, 2010;Trafimow, Hunt, Rice, & Geels, 2011;Trafimow & Rice, 2008, 2009. It is important to note that the definition of consistency, as it is used by PPT, may not be the same as definitions used in other areas of research.…”

“…(For a more detailed explanation and formal mathematical proofs, see Trafimow & Rice, 2008, 2009 It is then necessary to convert the corrected correlation back into cell frequencies using Equations A3-A6.…”