The Unpredictability of Predictability

Abstract

Like the alchemists of olden days looking for techniques to change base metals into gold, the psychologists of today are constantly searching for methods that will increase the validity of their predictions. Periodically a new approach is introduced and shows some promise---e.g., pure factor tests, pattern analysis and configural scoring, discriminant analysis--but usually when subjected to crossvalidation and replication these techniques have proven to be no more accurate than the regression model.Recently the problem has been approached from yet another angle, by attempting to predict predictability. Rather than assuming prediction errors are random, this approach postulates that there are systematic differences between predictable and unpredictable individuals. If a third (moderator) variable can be found which correlates with the degree of predictability (deviation, in standard score units, of the predicted criterion scores from the actual criterion score) a subgroup of predictable individuals can be identified. Then applying the predictor(s) only to individuals who are predictable the validity of the selection procedure can be increased. A side benefit is that the amount of testing is reduced (only predictable persons are given the complete battery), but this is accomplished at the expense of a reduction in the size of the pool of potential employees.Frederiksen and Melville (1954) using this basic technique showed that prediction could be improved by sub-grouping. They used "compulsivity", as measured by the Accountant scale on the Strong Vocational Interest Blank, as the predictability measure. A replication by Frederiksen and Gilbert (1960) provided partial confirmation of these results. Saunders (1956) using Frederiksen and Melville's data, plus other data, confirmed the earlier results. In addition, he provided a mathematical rationale for the problem and coined the term "moderator variable" to denote the measure of predictability.Ghiselli, in a series of studies, has shown that selection can be improved by screening on a predictability test, even if the moderator variable has a negligible zero-order validity coefficient (GhiseUi 1956); that a single test could serve both as a predictor and as a predictability measure (Ghiselli 1960a); and that a predictability measure could be developed that will tell which of two predictor tests makes the best prediction for a given individual. (Ghiselli 1960b).In a summary article Ghiselli (1963) concluded that there is substantial evidence of the moderator effect, that there is no general law to identify moderators --they are situation-specific, that the operation of moderators is not the same as 1These studies were conducted while Mr.

A failure to replicate the Bem and Allen study of individual differences in cross-situational consistency.

“…This assumption induced a plethora of investigators to incorporate self-report measures of consistency into their studies (e.g., LaPointe & Harrel, 1978;Lippa & Mash, 1981;Tunnell, 1980;Turner, 1978;Turner & Gilliam, 1979;Vestewig, 1978;Zanna, Olson, & Fazio, 1980), because the discovery of a general moderator variable would represent a major advance in the field of personality assessment (Fiske, 1957;Ghiselli, 1956). The results of our study clearly undermine this conclusion, and thus we must add our present results to the long list of failures in the search for moderators (e.g., Brown & Scott, 1966, 1967Goldberg, 1972;Kellogg, 1968;Strieker, 1966;Wallach & Leggett, 1972).…”

The Next Generation of Moderator Research in Personality Psychology

“…Perhaps because of this, these moderator effects could seldom be replicated and Ghiselli (1963) eventually concluded that efforts to improve predictive validity should be concentrated on other approaches. For a decade or so moderator variables served primarily as prey for psychometricians, who described a variety of methodological and statistical problems with moderator variable research (e.g.. Brown & Scott, 1966, 1967Goldberg, 1969;Kellogg, 1968;Strieker, 1966;Velicer, 1972).…”

Differential Predictability in College Admissions Testing

“…Thus the efficacy of the moderator model has yet to be demonstrated in the typical academic prediction situation (cf. Brown and Scott, 1966). The main problem seems to be identifying stable moderators, ones whose effect will hold up when subjected to cross-validation.…”