Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessments

Abstract

Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada's National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

“…Moreover, adjusting for examiner variance altered the pass/fail decisions of 11% of students in their study. Other studies have suggested that examiners' scores may be biased depending on the timing of OSCEs (Hope & Cameron, 2015;McLaughlin, Ainslie, Coderre, Wright, & Violato, 2009), the performance of other candidates (Yeates, Moreau, & Eva, 2015) or by different geographical locations (Sebok, Roy, Klinger, & De Champlain, 2015). Consequently, it is not sufficient to simply conduct an OSCE, and believe that the resulting scores are a fair representation of students' performance given the known influences of construct irrelevant variance.…”

Does Making the Numerical Values of Verbal Anchors on a Rating Scale Available to Examiners Inflate Scores on a Long Case Examination?

“…Although examiner differences exist and can contribute to construct-irrelevant variance within an examination, their variability as a group remains relatively consistent over time. 23 Based on the generalizability study, the comparatively similar estimates of the error variances due to examiner effects between the two years led us to believe that no systematic differences in examiners existed across cohorts.…”

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

“…Examples include use of item fit analysis to test for variability in item difficulty across schools or styles of test administration [29,30] and for evidence of unidimensionality in the scores forthcoming from a test of clinical competence. [31] Further examples include testing for rater leniency or harshness as determinants of student performance.…”

“…[31] Further examples include testing for rater leniency or harshness as determinants of student performance. [32,33] In some of the above cases, [30] …”