Examiner effect on the objective structured clinical exam – a study at five medical schools

Abstract

BackgroundThe Objective Structured Clinical Examination (OSCE) is increasingly used at medical schools to assess practical competencies. To compare the outcomes of students at different medical schools, we introduced standardized OSCE stations with identical checklists.MethodsWe investigated examiner bias at standardized OSCE stations for knee- and shoulder-joint examinations, which were implemented into the surgical OSCE at five different medical schools. The checklists for the assessment consisted of part A for knowledge and performance of the skill and part B for communication and interaction with the patient. At each medical faculty, one reference examiner also scored independently to the local examiner. The scores from both examiners were compared and analysed for inter-rater reliability and correlation with the level of clinical experience. Possible gender bias was also evaluated.ResultsIn part A of the checklist, local examiners graded students higher compared to the reference examiner; in part B of the checklist, there was no trend to the findings. The inter-rater reliability was weak, and the scoring correlated only weakly with the examiner’s level of experience. Female examiners rated generally higher, but male examiners scored significantly higher if the examinee was female.ConclusionsThese findings of examiner effects, even in standardized situations, may influence outcome even when students perform equally well. Examiners need to be made aware of these biases prior to examining.Electronic supplementary materialThe online version of this article (doi:10.1186/s12909-017-0908-1) contains supplementary material, which is available to authorized users.

Ethnic and Gender Biases in Clinical Performance Assessment (CPA) in Healthcare Education: A Systematic Review

“…In addition to potential ethnic-related bias affecting student grades, the effect of gender on the clinical performance of students is also a major concern [50,51]. Female students are more likely to perform better on communication and interpersonal skills than males [52][53][54][55] as well as on the overall performance of both written tests and clinical skills [27,35,44,51,56]. Furthermore, studies reported that medical students who showed empathy received better clinical evaluations, with women receiving higher scores on empathy scales than men [57].…”

“…Five selected studies published from 2007 to 2017 used retrospective cross-sectional design [3,27,35,44,56]. Adamson et al [68], Schleicher et al [51], and Wass et al [37] utilized prospective cross-sectional designs and Yeates et al [39] used a randomized controlled trial (RCT). The temporal distribution of the studies shows a progression from cross-sectional designs with the earliest studies to an RCT design with the most recent study.…”

“…Denney et al [35] collected data on the MRCGP CSA in the UK, Dewhurst et al [27] and McManus et al [3] recruited participants from the MRCP (UK), and Richens et al [56] included participants from the Intercollegiate Specialty Board examinations in the UK; these four studies were considered as representative of the population as they included all students for the speci ed range of time. Schleicher et al [51], Stupart et al [44], Wass et al [37], and Yeates et al [39] conducted convenience sampling at ve German medical schools, a medical school in South Africa, and medical schools throughout the UK, respectively; Adamson et al [68] used convenience sampling to recruit participants from the faculty of nursing faculty at a university in the United States.…”

“…McManus et al [3] most clearly described the reliability and validity of their outcome measures. Schleicher et al [51] addressed the weakness of the interrater reliability of their scale. Wass et al [37] and Yeates et al [39] did not describe the reliability of their rating scales.…”

“…Five selected studies were implemented in undergraduate medical education [3,27,35,37,39]; three studies in surgical residencies [44,51,56], and one study in nursing [68]. All studies provided prior training to examiners before the CPA or OSCE.…”