Stylistic Components of Human Judgment: The Generality of Individual Differences

A review and analysis of research on the test-retest reliability of professional judgment

“…In the ®rst (second) replicate, mean test±retest reliability was 0.76 (0.91) when task predictability was 0.80, but was only 0.59 (0.78) when task predictability was 0.40. Ramanaiah and Goldberg (1977) reported similar results for task-predictability levels of 0.80 ( prediction of freshman grade point averages) and 0.50 ( prediction of extent of involvement in campus leadership activities). Mean reliability of student subjects was 0.87 and 0.65 for the high-and low-predictability tasks, respectively.…”

“…There are two distinct types of test±retest reliability, which have been labeled`internal consistency' and temporal stability' (see e.g. Ramanaiah and Goldberg, 1977;Balzer, Rohrbaugh and Murphy, 1983;Cooksey, 1996). Internal consistency is assessed by a partial or full replication of judgments in a single task session.…”

“…Three studies have examined the eects on test±retest reliability of mechanically combining cues and replacing the judge with his or her own model. Ramanaiah and Goldberg (1977), in the study described above, constructed multiple regression models of their subjects' judgments Ð for both the initial and repeat judgments Ð in the grade-point and leadership prediction tasks, as well as in two additional tasks involving judgments of intelligence and sociability. The mean reliability of the subjects' actual judgments (regression-model-based predictions) for the grade-point, leadership, intelligence and sociability tasks was 0.87 (0.98), 0.65 (0.93), 0.90 (0.98) and 0.73 (0.95), respectively.…”

“…Cooksey, Freebody and Bennett (1990) had students rate the spelling diculty of 100 words, and repeated the task four weeks later. Multiple regression models were constructed for each student on each of the two occasions and, as in Ramanaiah and Goldberg (1977), used to predict the students' judgments. The median reliability of the actual judgments (regression-model-based predictions) was 0.59 (0.74).…”