Test–Retest Reliability of Ratings of Job Performance Dimensions in Managers

Abstract

This article examines the test-retest reliability of supervisory ratings for several dimensions of job performance and for overall job performance. We found that the test-retest reliability of overall job performance is .79 (SD .08), a value very close to the one found by Viswesvaran, Ones and Schmidt (1996), and that the average test-retest reliability for specific dimensions of job performance is .57 (SD .07). We also found that some dimensions of job performance appear to be easier to rate than others. We suggest some implications of these findings for research and practice of personnel selection.

Alternate Approaches to Understanding the Psychometric Properties of Assessment Centers: An analysis of the structure and equivalence of exercise ratings

“…Specifically, the dimension that was not invariant to the exercise, Sensitivity, is interpersonal in nature. This finding is consistent with prior research stemming from the performance rating literature suggesting that task‐ or problem‐solving‐related dimensions are rated with less error (based on interrater, intrarater, and rate–rerate reliability) and with less difficulty than dimensions that are more interpersonal in nature (Salgado, Moscoso, & Lado, ; Viswesvaran, Ones, & Schmidt, ; Viswesvaran, Schmidt, & Ones, ; Viswesvaran, Schmidt, & Ones, ). Viswesvaran et al.…”

In‐basket Validity: A systematic review

“…For criterion reliability, Pearlman et al's estimate had an average of .60. We note that Viswesvaran et al () identified .52 as a more accurate estimate and Salgado et al () obtained a sample size weighted mean reliability of .57 across performance dimensions. We used the Pearlman et al's distribution because: (a) they provided a distribution of criterion reliabilities that enabled both mean and variance corrections to be made and (b) the average of .60 would yield a more conservative estimate of validity than .52 or .57.…”

“…They found that supervisory ratings appeared to have higher reliability than peer ratings, and that the mean interrater reliability of supervisory ratings was .52 (using 40 reliability coefficients and a total sample of 14,650 participants) for overall job performance. Salgado, Moscoso, and Lado () found levels of test–retest reliability of job performance ratings across performance dimensions (.57) that were comparable with Viswesvaran and colleagues.…”

“…We relied on Pearlman, Schmidt, & Hunter's () assumed distribution of proficiency criterion reliabilities, in which the expected value is .60 (Pearlman et al's table ). This may provide an overestimate of reliability as later research estimated supervisor ratings reliability to be approximately .57 or .52 (Salgado et al, ; Viswesvaran et al, ). Thus, our estimates of the operational validity are likely conservative (i.e., downwardly biased).…”

Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment.

“…Apparently, consultants had the most difficulty rating the cluster of competencies related to personal management, as indicated by their lower interrater reliabilities. A potential explanation might be that these competencies are possibly less observable (Salgado, Moscoso, & Lado, 2003). Of course, conclusions with regard to the extent to which specific competencies can be reliably rated are bound by the fact that consultants rated only one job.…”