Do You See What I See? An Exploration of Congruence in Ratings From Multiple Perspectives¹

Abstract

This paper provides a comparative analysis of behavioral observations made on 152 service providers in a business advisory and professional services firm from 5 distinctly different ratings sources (self, direct reports, peers, supervisor, and clients). Results focused on differences in ratings level and degree of congruence with self‐assessments by observer type. The data suggested that service providers and their clients may have a different perceptual frame of reference than do internal observers (e.g., direct reports, peers, and supervisors). Moreover, congruence in self‐others' ratings was found to be a significant predictor of performance assessments from the same observer source. Implications of these results for the use of multirater appraisal systems are discussed.

The Evolution and Devolution of 360° Feedback

“…Although there is considerable research available discussing trends in rater group differences (e.g., Church, 1997Church, , 2000Church & Waclawski, 2001c;Furnham & Stringfield, 1994;Harris & Schaubroeck, 1988;J. W. Johnson & Ferstl, 1999), these studies presume that these groups can be meaningfully combined.…”

Hold the Line: An Examination of Line vs. Staff Differences

“…This was computed for each type of self-other comparison (self-direct report, selfpeer and self-supervisor) using a standard index of profile similarity-i.e., the square root of the sum of squared differences between self and other for each behavioral item divided by the total number of items on the instrument (e.g., Church, 1997a;1997b;Nunnally, 1978;Tisak & Smith, 1994). Often referred to in the literature as d (Edwards, 1994;Nunnally, 1978), this index of profile similarity was used because it reflects the total degree of congruence on an average item-by-item level rather than an already aggregated total score (Church, 1994(Church, , 1997aTisak & Smith, 1994).…”

“…More specifically, the day-to-day behavioral strengths and limitations of a number of line managers versus staff professionals in a large, rapidly growing United States (US) based highly service-oriented retail organization will be compared and contrasted using data collected from a multirater feedback initiative. In addition, given that recent research efforts with managers from a variety of organizations and industries (e.g., Atwater & Yammarino, 1992;Church, 1997aChurch, , 1997bFurnham & Stringfield, 1994) have supported the contention that higher performers have greater levels of managerial self-awareness-as measured by congruence in self-other ratings-the following study also attempts to address differential levels of ratings agreement across the two functions.…”

Foundation programme assessment tools in psychiatry

“…It can also encourage unduly positive feedback in the 'space for comments' section, owing to concerns that trainees may recognise anonymous comments highlighting poor performance. The most valid source of ratings for 'humanistic' dimensions are patients (Church, 1997), particularly in psychiatry, but they are surprisingly excluded from the assessment process.…”