Inter-Rater Reliability: Comparison of Checklist and Global Scoring for OSCEs

Abstract

Objective Structured Clinical Examinations (OSCEs) have been used globally in evaluating clinical competence in the education of health professionals. Despite the objective intent of OSCEs, scoring methods used by examiners have been a potential source of measurement error affecting the precision with which test scores are determined. In this study, we investigated the differences in the inter-rater reliabilities of objective checklist and subjective global rating scores of examiners (who were exposed to an online training program to standardise scoring techniques) across two medical schools. Examiners’ perceptions of the e-scoring program were also investigated. Two Australian universities shared three OSCE stations in their end-of-year undergraduate medical OSCEs. The scenarios were video-taped and used for on-line examiner training prior to actual exams. Examiner ratings of performance at both sites were analysed using generalisability theory. A single facet, all random persons by raters design [PxR] was used to measure inter-rater reliability for each station, separate for checklist scores and global ratings. The resulting variance components were pooled across stations and examination sites. Decision studies were used to measure reliability estimates. There was no significant mean score difference between examination sites. Variation in examinee ability accounted for 68.3% of the total variance in checklist scores and 90.2% in global ratings. Rater contribution was 1.4% & 0% of the total variance in checklist score and global rating respectively, reflecting high inter-rater reliability of the scores provided by co-examiners across the two schools. Score variance due to interaction and residual error was larger for checklist scores (30.3% vs 9.7%) than for global ratings. Reproducibility coefficients for global ratings were higher than for checklist scores. Survey results showed that the e-scoring package facilitated consensus on scoring techniques. This approach to examiner training also allowed examiners to calibrate the OSCEs in their own time. This study revealed that inter-rater reliability was higher for global ratings than for checklist scores, thus providing further evidence for the reliability of subjective examiner ratings

Use of Emergency Medicine Milestones as Items on End-of-Shift Evaluations Results in Overestimates of Residents' Proficiency Level

“…Compared with nominal scales, global rating scales (often utilizing a Likert scale) have been shown to better discriminate between different levels of training and have superior interrater reliability. 4,7,8 Furthermore, the consistency and accuracy of subjective global rating scores can be significantly improved with examiner training. 8 Given our findings and those of previous studies, a better assessment tool may be one that uses global ratings not based on PGY level and incorporates extensive faculty training.…”

Metodologias Ativas em Fisioterapia: Estudo de Confiabilidade Interexaminador do Método Osce

“…Ainda uma última hipótese pode servir para as duas estações e reside na existência da opção "insuficiente" como resposta no instrumento de avaliação da estação, que inclusive pode ser potencializada pela escassez de sistematização da prática profissional avaliada (estação 1) e pela diferença na formação/engajamento na elaboração do Osce (estação 4). No entanto, estudos recentes têm utilizado o global rating scale em substituição aos checklists e têm observado maior confiabilidade interexaminador 27 . Aquele possui, geralmente, mais de quatro opções de resposta no instrumento de avaliação, diferentemente dos checklists, que são instrumentos dicotomizados.…”

Addressing the ‘forgotten art of fundoscopy’: evaluation of a novel teaching ophthalmoscope

“…13 The assessors were also asked to independently provide a 'global impression', indicating whether the student was at a level that satisfied, exceeded, or did not satisfy expectations of a graduating medical student performing fundus examination of a normal eye. 28,29 During the second 'OSCE station', all students, regardless of their study arm, were required to use the teaching ophthalmoscope to examine the fundus of a previously unencountered 'patient' with a dilated pupil (using 1% tropicamide). Two additional masked assessors were employed with equivalent training to those in the first 'station'.…”