Trait, Rater and Level Effects in 360‐degree Performance Ratings

Abstract

Method and trait effects in multitrait-multirater (MTMR) data were examined in a sample of 2,350 managers who participated in a developmental feedback program. Managers rated their own performance and were also rated by two subordinates, two peers, and two bosses. The primary purpose of the study was to determine whether method effects are associated with the level of the rater (boss, peer, subordinate, self) or with each individual rater, or both. Previous research which has tacitly assumed that method effects are associated with the level of the rater has included only one rater from each level; consequently, method effects due to the rater's level may have been confounded with those due to the individual rater. Based on confirmatory factor analysis, the present results revealed that of the five models tested, the best fit was the 10-factor model which hypothesized 7 method factors (one for each individual rater) and 3 trait factors. These results suggest that method variance in MTMR data is more strongly associated with individual raters than with the rater's level. Implications for research and practice pertaining to multirater feedback programs are discussed.Multirater or 360-degree feedback systems are characterized by the evaluation of an individual's performance by multiple raters from multiple levels. Although procedures vary, typically the individual is rated by others who interact frequently with the individual, who are knowledgeable about the individual's performance, and whose opinions are valued by the individual. The most common procedure is to include peers, subordinates, and bosses (in addition to self-ratings), but raters outside the organization, such as customers or suppliers, may also be included.Multirater feedback programs differ from traditional appraisal programs in several ways. Aside from the use of multiple raters, multirater systems are used most frequently to enhance personal development and growth, rather than to help with salary administration, promotions, or COPYRIGHT 0 1998 PERSONNEL PSYCHOLOGY, INC.

Comparing Rater Groups: How To Disentangle Rating Reliability From Construct-Level Disagreements

“…Mount, Judge, Scullen, Sytsma, & Hezlett, 1998;Viswesvaran et al, 2002). Thus, although this process is plausible, it is not consistent with the empirical evidence.…”

“…Bracken et al (p. 765), despite academic research (Mount et al, 1998) clearly showing that it might be better from a true-score measurement perspective to combine ratings from naturally occurring work groups (peers, supervisors, etc. ), argue that "a true 360°feedback assessment, under our definition, must be designed in such a way that differences in rater perception are clearly identified and meaningful comparisons can be made between perceptions of different rater groups, agreement (or lack thereof) within a rater group, and even individual raters (e.g., self, manager) where appropriate.…”

“…Against this backdrop, however, empirical data have accumulated (e.g., Mount et al, 1998;Viswesvaran et al, 2002) to show that peers and supervisors are rating the same underlying constructs and that the observed between group correlations are attenuated due to interrater unreliability in peer and supervisory ratings (Viswesvaran et al, 1996(Viswesvaran et al, , 2002. Facteau and Craig (2001), using confirmatory factor analysis (CFA) and item response theory (IRT) with a large dataset of 360°feedback ratings, found construct equivalence between peers, supervisors, and subordinates.…”

The relationship between developmental performance appraisal, organizational support, organizational commitment and task performance: Testing a mediation model

“…Moreover, since employees' performance appraisal causes some conflict and opposition between administrators and employees, leading to the employees' exhibiting some deviant behaviors, supervisors treat them leniently, giving them high scores in order to prevent such conflicts (Spector, 2002;Decenzo and Robbins, 1996). Along with this issue, approximately 90 years ago, Thorndike (1920 cited in Mount et al,1998) observed that whenever administrators evaluate their employees' performances, the correlation between dimensions of work performance was higher that what was in reality. The studies conducted on this issue have shown that the effects of performance appraisal method are one the most major sources of error in rating performances.…”

360‐degree feedback with systematic coaching: Empirical analysis suggests a winning combination

“…In particular, research based on observational studies of managerial work (Kotter, 1982;Luthans, 1988;Mintzberg, 1973) established several managerial functions (e.g., planning, controlling), and interpersonal activities such as communicating and getting input from and providing feedback to others. Using Mann's (1965) conceptualizations, research examining the ratings of managerial effectiveness (e.g., Mount, Judge, Scullen, Sytsma, & Hezlett, 1998) identified three competence categories: administrative, human relations, and technical. Finally, social cognitive theory elaborates on the managerial importance of self-regulatory mechanisms such as selfawareness (Bandura, 1999;Stajkovic, 2003;Wood & Bandura, 1989).…”