Managing extremes of assessor judgment within the OSCE

Abstract

Rogue or rational? Investigating the impact of the extremes of assessor judgement within the OSCE. Abstract ContextThere is a growing body of research investigating assessor judgements in complex performance environments such as OSCE examinations. Post-hoc analysis can be employed to identify some elements of unwanted assessor variance. However, the impact of individual, apparently extreme O"CE pass/fail decisions has not been previously explored, This paper uses a range of studies as examples to illustrate O"CE and gives pragmatic suggestions to successfully alleviating problems. Method & ResultsWe used real OSCE assessment data from a number of examinations where at station level, a single examiner assesses student performance using a global grade and a key features checklist. Three exemplar case studies where initial post hoc analysis has indicated problematic individual assessor behaviour are considered and discussed in detail, highlighting both the impact of individual examiner behaviour and station design on subsequent judgments. ConclusionsIn complex assessment environments, institutions have a duty to maximise the defensibility, quality and validity of the assessment process. A key element of this involves critical analysis, through a range of approaches, of assessor judgments. However, care must be taken when assuming that apparent aberrant examiner behaviour is automatically just that.

The sights and insights of examiners in objective structured clinical examinations

“…At the ends of the examiner leniency distribution curve lie the ‘extreme’ assessors, defined as individuals giving a mean score greater or less than 3 standard deviations above or below the collective mean score [ 23 ]. The extreme nature of their assessments may be due to individual characteristics of an examiner, or less commonly, simple marking errors, for example grading 1/5 as ‘excellent’ and 5/5 as ‘fail’ when the opposite is correct [ 8 ].…”

“…Whilst OSCEs vary in their specific requirements and process across jurisdictions, the overall design of the OSCE has traditionally been viewed as advantageous, as it standardises the items and tasks for each candidate. Consequently, it has also been considered to minimise the effects of examiner bias through the use of ‘identical’ patients, structured checklists, and multiple assessor-candidate interactions across a number of stations [ 1 , 8 ]. Despite the intention of this design, OSCEs are in practice prone to high levels of variance [ 9 ].…”

Developing a video‐based method to compare and adjust examiner effects in fully nested OSCEs

“…Equally, as reliability is often influenced more by station specificity than by examiner variability, increasing the number of stations is likely to produce larger increases in reliability than examiner-focused approaches. 8 Conversely, many medical schools run OSCEs across multiple geographically dispersed sites, 18,46 in which the examiners at each site are drawn from clinicians who practise locally and who rarely interact with clinicians from other sites. In this (very common) instance it is reasonable to suggest that examiner cohorts could be systematically different in their practice norms and beliefs, the cohorts of trainees to whom they are exposed, their specialty mixes and their level of specialisation.…”

Using the Many-Facet Rasch Model to analyse and evaluate the quality of objective structured clinical examination: a non-experimental cross-sectional design

“…In the field of medical education, studies have mostly been conducted using classical test theory (CTT), examples have included: evaluating the effect of ‘extreme examiners’,8 the effect of contrast error,9 examiner leniency at the start of an OSCE session10 and standard setting 11. Although CTT is an often used and well-established model, item response theory (IRT) may offer some advantage.…”