Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

Yeates, Peter; Sebok‐Syer, Stefanie S

doi:10.1080/0142159x.2017.1248916

Cited by 22 publications

(26 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is a longstanding interest on the effect of examiners on standards in performance assessments like OSCEs (Bartman et al 2013;Downing 2005;Fuller et al 2017;Harasym et al 2008;Jefferies et al 2007;McManus et al 2006;Pell et al 2010;Yeates et al 2018;Yeates and Sebok-Syer 2017). Variation in examiner stringency is usually conceptualised and measured based on the scores (or grades) that examiners produce within stations-i.e.…”

Section: Examiner Stringency As An Effect On Scoresmentioning

confidence: 99%

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Homer

2020

Adv in Health Sci Educ

View full text Add to dashboard Cite

Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/ domain scores and global grades acting in combination. This complexity requires a more nuanced view of what stringency might mean when considering sources of variation of cut-scores in stations. This study uses data from 349 administrations of an 18-station, 36 candidate single circuit OSCE for international medical graduates wanting to practice in the UK (PLAB2). The station-level data was gathered over a 34-month period up to July 2019. Linear mixed models are used to estimate and then separate out examiner (n = 547), station (n = 330) and examination (n = 349) effects on borderline regression cut-scores. Examiners are the largest source of variation in cut-scores accounting for 56% of variance in cut-scores, compared to 6% for stations, < 1% for exam and 37% residual. Aggregating to the exam level tends to ameliorate this effect. For 96% of examinations, a 'fair' cutscore, equalising out variation in examiner stringency that candidates experience, is within one standard error of measurement (SEM) of the actual cut-score. The addition of the SEM to produce the final pass mark generally ensures the public is protected from almost all false positives in the examination caused by examiner cut-score stringency acting in candidates' favour.

show abstract

Section: Examiner Stringency As An Effect On Scoresmentioning

confidence: 99%

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Homer

2020

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…Equally, as reliability is often influenced more by station specificity than by examiner variability, increasing the number of stations is likely to produce larger increases in reliability than examiner-focused approaches. 8 Conversely, many medical schools run OSCEs across multiple geographically dispersed sites, 18,46 in which the examiners at each site are drawn from clinicians who practise locally and who rarely interact with clinicians from other sites. In this (very common) instance it is reasonable to suggest that examiner cohorts could be systematically different in their practice norms and beliefs, the cohorts of trainees to whom they are exposed, their specialty mixes and their level of specialisation.…”

Section: Implications Of Findingsmentioning

confidence: 99%

“…They found that site differences variably explained between 1.5% and 17.1% of score variability. Yeates and Sebok‐Syer specifically addressed whether parallel examiner cohorts across different sites in the same medical school showed different standards of judgement. Their provisional results suggested that scores by different examiner cohorts differed by up to 4.4% of the assessment scale.…”

Section: Introductionmentioning

confidence: 99%

Developing a video‐based method to compare and adjust examiner effects in fully nested OSCEs

et al. 2018

Self Cite

View full text Add to dashboard Cite

Background Although averaging across multiple examiners’ judgements reduces unwanted overall score variability in objective structured clinical examinations ( OSCE ), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner‐cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner‐cohort effects on students’ scores. Methods We developed video‐based examiner score comparison and adjustment ( VESCA ): volunteer students were filmed ‘live’ on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station‐specific common‐comparator videos, producing partial crossing between examiner cohorts. Many‐facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner‐cohort effects on students’ scores. Results After accounting for students’ ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students’ global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6–9.5% students’ scores were altered by at least 0.5 standard deviations of student ability. Conclusions Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness.

show abstract

“…These studies from educational measurement provide early approaches for assessing skills such as collaboration in ways that capture aspects of both independent and interdependent dimensions of performance and characterise performance along a spectrum rather than creating a false dichotomy. Within medical education, a variation of the Rasch measurement model proposed by Wilson and colleagues has already been used to capture aspects of rater's collaborative performance . All of these aforementioned approaches require us to consider the collective in order to meaningfully assess authentic clinical performance.…”

Section: Discussionmentioning

confidence: 99%

Considering the interdependence of clinical performance: implications for assessment and entrustment

et al. 2018

View full text Add to dashboard Cite

IntroductionOur ability to assess independent trainee performance is a key element of competency‐based medical education (CBME). In workplace‐based clinical settings, however, the performance of a trainee can be deeply entangled with others on the team. This presents a fundamental challenge, given the need to assess and entrust trainees based on the evolution of their independent clinical performance. The purpose of this study, therefore, was to understand what faculty members and senior postgraduate trainees believe constitutes independent performance in a variety of clinical specialty contexts.MethodsFollowing constructivist grounded theory, and using both purposive and theoretical sampling, we conducted individual interviews with 11 clinical teaching faculty members and 10 senior trainees (postgraduate year 4/5) across 12 postgraduate specialties. Constant comparative inductive analysis was conducted. Return of findings was also carried out using one‐to‐one sessions with key informants and public presentations.ResultsAlthough some independent performances were described, participants spoke mostly about the exceptions to and disclaimers about these, elaborating their sense of the interdependence of trainee performances. Our analysis of these interdependence patterns identified multiple configurations of coupling, with the dominant being coupling of trainee and supervisor performance. We consider how the concept of coupling could advance workplace‐based assessment efforts by supporting models that account for the collective dimensions of clinical performance.ConclusionThese findings call into question the assumption of independent performance, and offer an important step toward measuring coupled performance. An understanding of coupling can help both to better distinguish independent and interdependent performances, and to consider revising workplace‐based assessment approaches for CBME.

show abstract

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

Cited by 22 publications

References 25 publications

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Developing a video‐based method to compare and adjust examiner effects in fully nested OSCEs

Considering the interdependence of clinical performance: implications for assessment and entrustment

Contact Info

Product

Resources

About