Rater training revisited: An updated meta‐analytic review of frame‐of‐reference training

Abstract

The present study updates Woehr and Huffcutt's (1994) rater training meta-analysis and demonstrates that frame-of-reference (FOR) training is an effective method of improving rating accuracy. The current meta-analysis includes over four times as many studies as included in the Woehr and Huffcutt meta-analysis and also provides a snapshot of current rater training studies. The present meta-analysis also extends the previous meta-analysis by showing that not all operationalizations of accuracy are equally improved by FOR training; Borman's differential accuracy appears to be the most improved by FOR training, along with behavioural accuracy, which provides a snapshot into the cognitive processes of the raters. We also investigate the extent to which FOR training protocols differ, the implications of protocol differences, and if the criteria of interest to FOR researchers have changed over time.The evaluation of individual work performance is a long-standing and ubiquitous concern for both organizational researchers and practitioners alike. To date, there have literally been thousands of published articles focusing on performance appraisal (PA) (a cursory search of SSCI using 'performance appraisal' as the search term revealed 2,887 hits). Moreover, an overwhelming proportion of this literature has focused on ways to improve the quality of evaluations provided by individual raters. In general, two approaches have been pursued with respect to improving performance ratings: developing better rating scales and training raters. Of these two approaches, rater training is by far the most successful and widely accepted. In fact, Landy and Farr (1980) called for a moratorium on research investigating rating scales because scale differences accounted for only a small percentage of the variance in ratings. On the other hand, training raters can result in large improvements in rating accuracy (e.g., Woehr & Huffcutt, 1994). Moreover, even though very little empirical research has investigated the benefits of rater training

Entrustability Scales

“…1 Indeed, evidence shows that rater training can increase proper use of an assessment tool 35 ; however, these training sessions are resource intensive, time consuming (often requiring two to four hours of a rater's time), and difficult to implement outside of a study environment. 36 For this reason, their potential long-term feasibility is questionable.…”

Preliminary Investigation of the Impact of a Web-Based Module on Direct Behavior Rating Accuracy

“…Using frame-of-reference training, participants review the performance of targets and then aspects of the performance central to evaluation are discussed relative to each performance dimension. Frame-ofreference training has been found to be effective in improving rating accuracy and is often the method of choice when conducting rater training (Athey & McIntyre, 1987;Roch, Woehr, Mishra, & Kieszczynska, 2012;Woehr & Huffcutt, 1994).…”

A test of the generalizability of a recently suggested conceptual model for assessment center ratings

“…Candidates were rated by two experienced assessors (a military officer trained in human resource management and a psychologist from the field of psychological assessment) who received at least one day of frame-of-reference training (Roch, Woehr, Mishra, & Kieszczynska, 2012). Following each exercise in the AC, every candidate was rated on three to six dimensions with a 4-point scale ranging from 1 (clearly failed to meet requirements) to 4 (clearly exceeded requirements).…”