Using cultural historical activity theory to reflect on the sociocultural complexities in OSCE examiners’ judgements

Abstract

Examiners’ judgements play a critical role in competency-based assessments such as objective structured clinical examinations (OSCEs). The standardised nature of OSCEs and their alignment with regulatory accountability assure their wide use as high-stakes assessment in medical education. Research into examiner behaviours has predominantly explored the desirable psychometric characteristics of OSCEs, or investigated examiners’ judgements from a cognitive rather than a sociocultural perspective. This study applies cultural historical activity theory (CHAT) to address this gap in exploring examiners’ judgements in a high-stakes OSCE. Based on the idea that OSCE examiners’ judgements are socially constructed and mediated by their clinical roles, the objective was to explore the sociocultural factors that influenced examiners’ judgements of student competence and use the findings to inform examiner training to enhance assessment practice. Seventeen semi-structured interviews were conducted with examiners who assessed medical student competence in progressing to the next stage of training in a large-scale OSCE at one Australian university. The initial thematic analysis provided a basis for applying CHAT iteratively to explore the sociocultural factors and, specifically, the contradictions created by interactions between different elements such as examiners and rules, thus highlighting the factors influencing examiners’ judgements. The findings indicated four key factors that influenced examiners’ judgements: examiners’ contrasting beliefs about the purpose of the OSCE; their varying perceptions of the marking criteria; divergent expectations of student competence; and idiosyncratic judgement practices. These factors were interrelated with the activity systems of the medical school’s assessment practices and the examiners’ clinical work contexts. Contradictions were identified through the guiding principles of multi-voicedness and historicity. The exploration of the sociocultural factors that may influence the consistency of examiners’ judgements was facilitated by applying CHAT as an analytical framework. Reflecting upon these factors at organisational and system levels generated insights for creating fit-for-purpose examiner training to enhance assessment practice.

Learner Experiences of Preceptor Self-Disclosure of Personal Illness in Medical Education

“…44,45 The use of activity theory as a sensitizing concept allowed us to forefront participants' experiences and ground our interpretations in the data. 46 Ensuring trustworthiness of the qualitative data…”

“…Following recommendations by Yamagata-Lynch, 43 patterns established via thematic analysis were subsequently opened and interrogated with attention to contradictions and expansive learning 44,45 . The use of activity theory as a sensitizing concept allowed us to forefront participants’ experiences and ground our interpretations in the data 46 …”

Better together: An assessor support roadmap

“…However, consistency of assessor judgements of learner performance has been a concern in directly observed clinical assessments such as workplace‐based assessments (WBAs) and objective structured clinical examinations (OSCEs) 1,2 . A range of sociocultural factors could influence the consistency of assessor judgements such as assessors' beliefs about the purpose of an assessment, their perception of the usefulness of the marking criteria, their expectations of learner competence and their idiosyncratic judgement practices 3 . These inconsistencies affect the high‐stakes decisions made regarding learner progression or feedback provided that could impact their career development.…”

“… 1 , 2 A range of sociocultural factors could influence the consistency of assessor judgements such as assessors' beliefs about the purpose of an assessment, their perception of the usefulness of the marking criteria, their expectations of learner competence and their idiosyncratic judgement practices. 3 These inconsistencies affect the high‐stakes decisions made regarding learner progression or feedback provided that could impact their career development.…”

“…1,2 A range of sociocultural factors could influence the consistency of assessor judgements such as assessors' beliefs about the purpose of an assessment, their perception of the usefulness of the marking criteria, their expectations of learner competence and their idiosyncratic judgement practices. 3…”