Corrections for criterion reliability in validity generalization: The consistency of Hermes, the utility of Midas

Abstract

a b s t r a c tThere is criticism in the literature about the use of interrater coefficients to correct for criterion reliability in validity generalization (VG) studies and disputing whether .52 is an accurate and non-dubious estimate of interrater reliability of overall job performance (OJP) ratings. We present a second-order meta-analysis of three independent meta-analytic studies of the interrater reliability of job performance ratings and make a number of comments and reflections on LeBreton et al.'s paper. The results of our meta-analysis indicate that the interrater reliability for a single rater is .52 (k = 66, N = 18,582, SD = .105). Our main conclusions are: (a) the value of .52 is an accurate estimate of the interrater reliability of overall job performance for a single rater; (b) it is not reasonable to conclude that past VG studies that used .52 as the criterion reliability value have a less than secure statistical foundation; (c) based on interrater reliability, test-retest reliability, and coefficient alpha, supervisor ratings are a useful and appropriate measure of job performance and can be confidently used as a criterion; (d) validity correction for criterion unreliability has been unanimously recommended by "classical" psychometricians and I/O psychologists as the proper way to estimate predictor validity, and is still recommended at present; (e) the substantive contribution of VG procedures to inform HRM practices in organizations should not be lost in these technical points of debate.

A Theoretical Model of Psychometric Effects of Faking on Assessment Procedures: Empirical findings and implications for personality at work

“…A third limitation is that the effects of faking were examined for the Cronbach's alpha coefficient of reliability. Probably, the effects of faking apply to other reliability coefficients such as the test–retest coefficient, the interrater coefficient, and the CES (Salgado, ; Salgado, Moscoso, & Anderson, ; Salgado & Moscoso, ; Schmidt, Le, & Ilies, ), but additional research is required.…”

Predicting expatriate effectiveness: The role of personality, cross‐cultural adjustment, and organizational support

“…The compound of expatriate effectiveness showed a Cronbach alpha of .90 in this sample. Unfortunately, we cannot obtain an estimate of the interrater reliability which is the most appropriate reliability coefficient for job performance ratings (Salgado, 2015;Salgado & Moscoso, 1996;Salgado, Moscoso, & Anderson, 2016;Viswesvaran et al, 1996).…”

Validez predictiva e impacto adverso de la entrevista conductual estructurada en el sector público

“…En ausencia de datos de fiabilidad interjueces -en esta investigación cada persona contratada fue evaluada únicamente por un evaluador-, se empleó la estimación metaanalítica de la fiabilidad interjueces, r xx = .52, como estimación de la fiabilidad del criterio (Viswesvaran et al, 1996). Pese a la visión de autores como Lebreton, Scherer y James (2014), que están en desacuerdo con esta corrección, que a su juicio infla artificialmente las correlaciones obtenidas tras la corrección, existe un consenso amplio basado en numerosas y potentes evidencias en que cuando se emplean escalas de evaluación el coeficiente de fiabilidad a considerar sea el de fiabilidad interevaluadores, cuya estimación es .52 en el caso de un solo evaluador (Sackett, 2014;Salgado, Moscoso y Anderson, 2016;Shen, Cucina, Walmsley y Seltzer, 2014;Viswesvaran, Ones, Schmidt, Le y Oh, 2014). Emplear el coeficiente alfa de Cronbach como estimación de la fiabilidad de la evaluación del desempeño no es una alternativa adecuada ya que solo controla una fuente de error (el error aleatorio), dejando de lado el error transitorio y el error debido a los diferentes evaluadores.…”

“…El coeficiente test-retest tampoco lo es, ya que no controla la falta de acuerdo entre evaluadores. Sin embargo, los juicios del supervisor son una media útil y adecuada del desempeño y pueden ser usados con confianza como criterio, especialmente cuando los datos de desempeño se recogen en el intervalo temporal de un año, como en esta investigación (Salgado, 2015;Salgado et al, 2016;Schmidt, Le e Ilies, 2003).…”