Seeing the same thing differently

Abstract

Assessors' scores in performance assessments are known to be highly variable. Attempted improvements through training or rating format have achieved minimal gains. The mechanisms that contribute to variability in assessors' scoring remain unclear. This study investigated these mechanisms. We used a qualitative approach to study assessors' judgements whilst they observed common simulated videoed performances of junior doctors obtaining clinical histories. Assessors commented concurrently and retrospectively on performances, provided scores and follow-up interviews. Data were analysed using principles of grounded theory. We developed three themes that help to explain how variability arises: Differential Salience-assessors paid attention to (or valued) different aspects of the performances to different degrees; Criterion Uncertainty-assessors' criteria were differently constructed, uncertain, and were influenced by recent exemplars; Information Integration-assessors described the valence of their comments in their own unique narrative terms, usually forming global impressions. Our results (whilst not precluding the operation of established biases) describe mechanisms by which assessors' judgements become meaningfully-different or unique. Our results have theoretical relevance to understanding the formative educational messages that performance assessments provide. They give insight relevant to assessor training, assessors' ability to be observationally "objective" and to the educational value of narrative comments (in contrast to numerical ratings).

Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance

“…Thus we cannot stipulate the magnitude of the effect our intervention may have had on the participants. However, our findings are in line with findings in recent studies (Govaerts et al 2011(Govaerts et al , 2013Kogan et al 2011;Yeates et al 2013). …”

“…Internal sources of information, which are unique to each individual rater, are also essential to the ability to form assessments. These C. St-Onge et al internal sources are in a constant state of flux as the rater acquires additional experiences with trainees and with other evaluators (a finding that is consistent with the observations from Kogan et al (2011) and Yeates et al (2013).…”

“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

“…Whether this challenge is addressed from a psychometric perspective, for example generalizability theory [1], or from social cognition frameworks [2–5], student grades should depend as little as possible on who examines them.…”

“…Several studies have shown that clinician examiners manifest different levels of severity, and that this has a significant impact on examinee grades and assessment decisions across a range of clinical skills assessments. These include work-based assessments, oral examinations and OSCEs [5–8]. Although extreme differences in examiner severity are probably relatively uncommon [9] modest differences in examiner severity may make important differences to student grades and pass-fail decisions.…”

Emotions and assessment: considerations for rater-based judgements of entrustment

“…Although a variety of approaches to assessment exist, the implementation of competency‐based education highlights the role of performance‐based assessment, particularly rater‐based assessment approaches that recognise the developmental trajectories of learners. In order to ensure the efficacy and defensibility of assessment, and to test the alignment between the intended uses and practices of assessment, recent work has shed light on the cognitive underpinnings and processes at play in rater‐based assessment . The majority of the broader literature on raters in health professions education (HPE) has focused on cognitive or contextual factors that influence rater judgements.…”