Challenges in using rater judgements in medical education

Abstract

Changes in the healthcare environment are putting increasing pressure on medical schools to make faculty accountable and to document the quality of the medical education they provide. Faculty's ratings of students' performances and students' ratings of faculty's teaching are important elements in these efforts to document educational quality. This article discusses selected research related to factors affecting raters' judgements, analyses how changes in the health care environment are influencing such judgements, offers some suggestions to moderate some of the effects and links these influences to the system that upholds professional standards. Ratings are known to have a positive bias (generosity error), provide limited discrimination and often fail to document serious deficits. The potential sources of these problems relate to the mechanics of the rating task, the system used to obtain ratings and factors affecting rater judgement. As managed care demands reduce the time faculty have for teaching, as system-wide disincentives to provide negative ratings proliferate and as social engineering challenges, such as the Americans with Disabilities Act, impose differential standards for students, the natural tendency to avoid giving negative ratings becomes even harder to resist. Ultimately, these forces compromise the capability of faculty to uphold the standards of the profession. The author calls for a national effort to stem the erosion of those standards.

Frameworks for learner assessment in medicine: AMEE Guide No. 78

“…Halo-effects and low intra-and interrater reliability are ubiquitous among untrained assessors of medical trainees (Albanese 2000;Williams et al 2003). This may in part be caused by a lack of a mental frame of reference (Holmboe et al 2011), but also by the complexity of the assessment task, or by the tendency of humans to categorize others in predefined groups.…”

De relatie tussen normen en oordelen in het toezicht op de gezondheidszorg

“…10 Een coöperatieve handhavingsstijl is gericht op het bewaken van de goede relatie tussen inspectee en inspecteur. 10 Problemen van een coöperatieve handhavingsstijl zijn het ''humaness-problem'', 12 ''client intimacy'',''capturing'' 13 en ''the shadow of the future''. 14 De inspecteur wordt door zijn betrokkenheid bij de onder toezicht gestelde, ingekapseld en oordeelt daardoor niet meer objectief, 13,15 of de inspecteur wil de relatie met de instelling niet schaden omdat ze ook in de toekomst nog samen verder moeten.…”

“…12 Implicaties en oplossingen validiteitsprobleem De IGZ gebruikt het GT om risicogericht toezicht te houden. Instellingen waar risico's in de zorg vermoed worden, bezoekt de IGZ om ter plaatse de kwaliteit van zorg in kaart te brengen en deze te stimuleren.…”

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall

“…[1][2][3][4] Another source of variability appears to be the format of the rating instrument. Efforts to improve the reliability of WBAs have generally been directed toward manipulating forms and rating scales.…”