Examiners’ decision‐making processes in observation‐based clinical examinations

Malau‐Aduli, Bunmi S.; Hays, Richard; D'Souza, Karen; Smith, Amy M.; Jones, Karina; Turner, Richard; Shires, Lizzi; Smith, Jane; Richmond, Cassandra; Celenza, Antonio; Gupta, Tarun Sen

doi:10.1111/medu.14357

Cited by 14 publications

(14 citation statements)

References 67 publications

(80 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 ). It seems there is little specific literature in this area, and it might be valuable to further investigate qualitatively the different processes of how and why examiners might award grades and scores differently (Malau-Aduli et al, 2021 ; Tavares & Eva, 2013 ). Under examinee-centred methods of standard setting (e.g.…”

Section: Discussionmentioning

confidence: 99%

Pass/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes

Homer

2022

Adv in Health Sci Educ

View full text Add to dashboard Cite

Variation in examiner stringency is a recognised problem in many standardised summative assessments of performance such as the OSCE. The stated strength of the OSCE is that such error might largely balance out over the exam as a whole. This study uses linear mixed models to estimate the impact of different factors (examiner, station, candidate and exam) on station-level total domain score and, separately, on a single global grade. The exam data is from 442 separate administrations of an 18 station OSCE for international medical graduates who want to work in the National Health Service in the UK. We find that variation due to examiner is approximately twice as large for domain scores as it is for grades (16% vs. 8%), with smaller residual variance in the former (67% vs. 76%). Combined estimates of exam-level (relative) reliability across all data are 0.75 and 0.69 for domains scores and grades respectively. The correlation between two separate estimates of stringency for individual examiners (one for grades and one for domain scores) is relatively high (r=0.76) implying that examiners are generally quite consistent in their stringency between these two assessments of performance. Cluster analysis indicates that examiners fall into two broad groups characterised as hawks or doves on both measures. At the exam level, correcting for examiner stringency produces systematically lower cut-scores under borderline regression standard setting than using the raw marks. In turn, such a correction would produce higher pass rates—although meaningful direct comparisons are challenging to make. As in other studies, this work shows that OSCEs and other standardised performance assessments are subject to substantial variation in examiner stringency, and require sufficient domain sampling to ensure quality of pass/fail decision-making is at least adequate. More, perhaps qualitative, work is needed to understand better how examiners might score similarly (or differently) between the awarding of station-level domain scores and global grades. The issue of the potential systematic bias of borderline regression evidenced for the first time here, with sources of error producing cut-scores higher than they should be, also needs more investigation.

show abstract

Section: Discussionmentioning

confidence: 99%

Pass/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes

Homer

2022

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…This mental "shortcut" is faster and reduces cognitive complexity when working memory is overloaded by time pressure or increased complexity and is also more likely in experienced assessors who recognize patterns more quickly (14,16,17). Applying this concept to our previous study, assessors appeared to use a representativeness heuristic to consider "how much does the observed clinical performance of a senior medical student compare with what I expect of a 'prototypical' intern" (12)? Such representational heuristics may be influenced by assessors' roles and experiences, contrast effects, use of inference, working memory effects, different interpretations of behaviors, predisposition to consider a particular perspective (e.g., of the learner or patient), different pre-existing frames of reference, exposure to different learner cohorts, and the examiners' own clinical skills and perceptions of task difficulty (18)(19)(20)(21).…”

Section: Introductionmentioning

confidence: 95%

“…This highlights a need for greater understanding of the cognitive processes of clinical assessors to inform strategies that enhance fair and robust judgements. Our previous research showed that judging candidate performance is complex, cognitively challenging and mentally demanding, particularly when borderline performance is observed in an "exit" Objective Structured Clinical Examination (OSCE) (12). In this "grey" zone of candidate performance, assessors used academic institutional marking criteria as a "safety blanket" to guide judgement, but also used additional criteria that were not necessarily explicit in the marking sheet, based on professional expectations (candidate demeanor and patient safety).…”

Section: Introductionmentioning

confidence: 99%

“Could You Work in My Team?”: Exploring How Professional Clinical Role Expectations Influence Decision-Making of Assessors During Exit-Level Medical School OSCEs

et al. 2022

Self Cite

View full text Add to dashboard Cite

Decision-making in clinical assessment, such as exit-level medical school Objective Structured Clinical Examinations (OSCEs), is complex. This study utilized an empirical phenomenological qualitative approach with thematic analysis to explore OSCE assessors' perceptions of the concept of a “prototypical intern” expressed during focus group discussions. Topics discussed included the concept of a prototypical intern, qualities to be assessed, and approaches to clinical assessment decision-making. The thematic analysis was then applied to a theoretical framework (Cultural Historical Activity Theory—CHAT) that explored the complexity of making assessment decisions amidst potentially contradicting pressures from academic and clinical perspectives. Ten Australasian medical schools were involved with 15 experienced and five less experienced assessors participating. Thematic analysis of the data revealed four major themes in relation to how the prototypical intern concept influences clinical assessors' judgements: (a) Suitability of marking rubric based on assessor characteristics and expectations; (b) Competence as final year student vs. performance as a prototypical intern; (c) Safety, trustworthiness and reliability as constructs requiring assessment and (d) Contradictions in decision making process due to assessor differences. These themes mapped well within the interaction between two proposed activity systems in the CHAT model: academic and clinical. More clinically engaged and more experienced assessors tend to fall back on a heuristic, mental construct of a “prototypical intern,” to calibrate judgements, particularly, in difficult situations. Further research is needed to explore whether consensus on desirable intern qualities and their inclusion into OSCE marksheets decreases the cognitive load and increases the validity of assessor decision making.

show abstract

“…In this context, assessor judgements are guided by prescribed expectations and scoring criteria, provided by assessment academics, outlined on the OSCE sheet. The judgements offered are not ‘objective’, however, as my colleagues and I have noted that the process of making judgements about student performance involves combining analytical and affective elements with an intuitive drive to rate candidates against a personal construct of a ‘prototypical intern’ 7 . In making such a comparison, the specific, time‐limited, and pre‐scripted nature of OSCE encounters may require assessors to adapt their expectations of learner performance to take into account situational constraints.…”

mentioning

confidence: 99%

“…The judgements offered are not 'objective', however, as my colleagues and I have noted that the process of making judgements about student performance involves combining analytical and affective elements with an intuitive drive to rate candidates against a personal construct of a 'prototypical intern'. 7 In making such a comparison, the specific, time-limited, and pre-scripted nature of OSCE encounters may require assessors to adapt their expectations of learner performance to take into account situational constraints. In a follow-up study, 8 we explored OSCE assessors' perceptions of the 'prototypical' construct by applying a theoretical framework (Cultural Historical Activity Theory [CHAT]) that enabled examination of the complexity inherent in making assessment decisions.…”

mentioning

confidence: 99%