Assessor training: its effects on criterion‐based assessment in a medical context

Abstract

Increasingly, academic institutions are being required to improve the validity of the assessment process; unfortunately, often this is at the expense of reliability. In medical schools (such as Leeds), standardized tests of clinical skills, such as Objective Structured Clinical Examinations (OSCEs) are widely used to assess clinical competence, both at the undergraduate and postgraduate levels. However, the issue of setting the pass marks or passing standard for such examinations remains contentious. The arrangements for particular OSCE assessment activities usually involve many different assessors, and have practical aspects that cannot be exactly duplicated across the cohort. These complexities therefore raise issues with respect to the robustness of the comparative student grading mechanism. This article addresses one aspect affecting the reliability of assessment, namely the effects of assessor training on the awarding of student marks. The article also investigates the interaction of gender between assessors, trained and untrained, and students. The findings, which are based on a detailed analysis of final year OSCE marks, indicate that untrained assessors award higher marks than trained assessors, and that a gender interaction exists; more specifically, that the use of untrained assessors tends to benefit female students over male students. The tension between reliability and validity has been particularly important in the field of medical education for a number of years -with medical students in the latter stages of their courses often being required to demonstrate competence in a variety of different simulated clinical activities with different patients, in front of different assessors in different hospitals and on different days. The complex nature of the OSCE arrangements raises serious questions as to the robustness of the setting the pass/fail boundary and, to a lesser extent, the honours boundaries. Overview What is already known on this subjectThere are differing opinions as to the effects of assessor training on the quality of judgements in the area of criterion-based assessment; either that it makes no overall difference to the degree of severity in the marking, or that it leads to increasing levels of stringency. What this study addsThis is a large study based on 207 students and 108 assessors. It quantifies the overall effects of assessor training with reference to the potential impact on student pass rates; it

How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49

“…For example, assessor training effects, staff / student gender effects, and associated interactions, which have all been previously described (Pell 2008), and which underline the need for complete and enhanced assessor training as previously highlighted (Holmboe 2004).…”

Developing a video‐based method to compare and adjust examiner effects in fully nested OSCEs

“…Examiner variability is often a substantial source of construct‐irrelevant variance in OSCEs . Training examiners is strongly recommended and some empirical findings support its benefits . In the original conceptions of OSCEs, all students were intended to meet all examiners, and as such examiner variability was unlikely to advantage or disadvantage particular students unless examiners showed idiosyncratic behaviour towards subsets of students .…”

Awarding global grades in OSCEs: Evaluation of a novel eLearning resource for OSCE examiners

“…There has been limited research regarding OSCE examiner training and the impact this has on their performance (Boursicot et al 2011). However, Pell et al did find that OSCE examiner training made a significant difference to the marks they awarded candidates (Pell et al 2008).…”