Reliability and Accuracy of Resident Evaluations of Surgical Faculty

Abstract

This study examines the reliability and accuracy of ratings by general surgery residents of surgical faculty. Twenty-three of 33 residents anonymously and voluntarily evaluated 62 surgeons in June, 1988; 24 of 28 residents evaluated 64 surgeons in June, 1989. Each resident rated each surgeon on a 5-point scale for each of 10 areas of performance: technical ability, basic science knowledge, clinical knowledge, judgment, peer relations, patient relations, reliability, industry, personal appearance, and reaction to pressure. Reliability analyses evaluated internal consistency and interrater correlation. Accuracy analyses evaluated halo error, leniency/severity, central tendency, and range restriction. Ratings had high internal consistency (coefficient alpha = 0.97). Interrater correlations were moderately high (average Pearson correlation = 0.63 among raters). Ratings were generally accurate, with halo error most prevalent and some evidence of leniency. Ratings by chief residents had the least halo. Results were generally replicable across the two academic years. We conclude that anonymous ratings of surgical faculty by groups of residents can provide a reliable and accurate evaluation method, ratings by chief residents are most accurate, and halo error may pose the greatest threat to accuracy, pointing to the need for greater definition of evaluation items and scale points.

Clinical supervisor evaluations during general surgery clerkships

“…performance evaluation gives similar ratings across all dimensions rather than distinguish amongst single dimensions (Risucci et al 1992;Williams et al 2003). In contrast to consultants, surgical residents and interns can spend significant amounts of time teaching and directly supervising medical students in a wide variety of different clinical settings (Lowry 1976;Pelletier & Belliveau 1999;Minor & Poenaru 2002;Whittaker et al 2006).…”

Assessing the reliability of student evaluations of teaching: choosing the right coefficient

“…Cohen, MacRae, and Jamieson (1996) found that medical students evaluating their surgery instructors achieved an intraclass correlation of only 0.65. Similarly, Risucci et al (1992) found that interrater correlations only averaged between 0.62 and 0.64. Donnelly and Woolliscroft (1989) found intraclass class reliabilities for 12 evaluation items ranged from a low of 0.64 to a high of 0.82.…”

Chief Resident Case Experience and Autonomy Are Associated With Resident Confidence and Future Practice Plans

“…Indeed, the judicious use of subjective instruments may be quite reliable, even if less standardized than a national examination, and previous subjective studies of surgeon evaluations have shown good internal consistency. 25,26 This applies to the present study as well as the internal measurements used within a program for resident education.…”