A method for identifying extreme OSCE examiners

Bartman, Ilona; Smee, Sydney; Roy, Marguerite

doi:10.1111/j.1743-498x.2012.00607.x

Cited by 29 publications

(34 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These include work-based assessments, oral examinations and OSCEs [5–8]. Although extreme differences in examiner severity are probably relatively uncommon [9] modest differences in examiner severity may make important differences to student grades and pass-fail decisions.…”

Section: Introductionmentioning

confidence: 99%

“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

et al. 2017

View full text Add to dashboard Cite

BackgroundRobust and defensible clinical assessments attempt to minimise differences in student grades which are due to differences in examiner severity (stringency and leniency). Unfortunately there is little evidence to date that examiner training and feedback interventions are effective; “physician raters” have indeed been deemed “impervious to feedback”. Our aim was to investigate the effectiveness of a general practitioner examiner feedback intervention, and explore examiner attitudes to this.MethodsSixteen examiners were provided with a written summary of all examiner ratings in medical student clinical case examinations over the preceding 18 months, enabling them to identify their own rating data and compare it with other examiners. Examiner ratings and examiner severity self-estimates were analysed pre and post intervention, using non-parametric bootstrapping, multivariable linear regression, intra-class correlation and Spearman’s correlation analyses. Examiners completed a survey exploring their perceptions of the usefulness and acceptability of the intervention, including what (if anything) examiners planned to do differently as a result of the feedback.ResultsExaminer severity self-estimates were relatively poorly correlated with measured severity on the two clinical case examination types pre-intervention (0.29 and 0.67) and were less accurate post-intervention. No significant effect of the intervention was identified, when differences in case difficulty were controlled for, although there were fewer outlier examiners post-intervention. Drift in examiner severity over time prior to the intervention was observed. Participants rated the intervention as interesting and useful, and survey comments indicated that fairness, reassurance, and understanding examiner colleagues are important to examiners.ConclusionsDespite our participants being receptive to our feedback and wanting to be “on the same page”, we did not demonstrate effective use of the feedback to change their rating behaviours. Calibration of severity appears to be difficult for examiners, and further research into better ways of providing more effective feedback is indicated.

show abstract

Section: Introductionmentioning

confidence: 99%

“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

et al. 2017

View full text Add to dashboard Cite

show abstract

“…The nature of these case studies (revealed through post hoc analysis) meant we were unable to perform direct observation of the assessor-candidate encounters to better understand the challenges of a complex performance assessment, nor undertake detailed O"CE Whilst the number of cases we are able to present in this work is small, the impact of these particularly for pass/fail decisions in the critical region, reflecting the findings of others (Bartman et al 2013). It is also tempting to postulate that such problems would not arise through the use of global grades only (as employed by many institutions), but we would argue that the use of key feature scoring/checklist formats and global grades allowed the generation of sophisticated station level metrics which consequently aided in detection and categorisation of assessor judgements judged to be at the extreme .…”

Section: Discussionmentioning

confidence: 75%

“…Small bodies of work have started to concentrate on identifying examiners that are at the extremes of marking at station level, revealing in one study that <0.3% of examiners can be classed as extreme [ three standard deviations beyond the mean of all raters] (Bartman et al 2013). Whilst the overall proportion in these case studies, it are a reflection of a complex environment which exerts powerful effects on assessment through test constructs and scoring formats and judgements about safe clinical care (Kogan et al 2014).…”

Section: Discussionmentioning

confidence: 99%

“…Sophisticated post-hoc analyses of the OSCE now allow us to identify variance due to non-student factors including those related to the delivery of the OSCE (e.g. OSCEs delivered across multiple centres, timing of parallel examination cycles), scoring rubrics and assessor judgements at overall test level, including (Pell et al 2010;Bartman et al 2013).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Managing extremes of assessor judgment within the OSCE

et al. 2016

View full text Add to dashboard Cite

Rogue or rational? Investigating the impact of the extremes of assessor judgement within the OSCE. Abstract ContextThere is a growing body of research investigating assessor judgements in complex performance environments such as OSCE examinations. Post-hoc analysis can be employed to identify some elements of unwanted assessor variance. However, the impact of individual, apparently extreme O"CE pass/fail decisions has not been previously explored, This paper uses a range of studies as examples to illustrate O"CE and gives pragmatic suggestions to successfully alleviating problems. Method & ResultsWe used real OSCE assessment data from a number of examinations where at station level, a single examiner assesses student performance using a global grade and a key features checklist. Three exemplar case studies where initial post hoc analysis has indicated problematic individual assessor behaviour are considered and discussed in detail, highlighting both the impact of individual examiner behaviour and station design on subsequent judgments. ConclusionsIn complex assessment environments, institutions have a duty to maximise the defensibility, quality and validity of the assessment process. A key element of this involves critical analysis, through a range of approaches, of assessor judgments. However, care must be taken when assuming that apparent aberrant examiner behaviour is automatically just that.

show abstract

“…We only retrospectively collected a limited amount of candidates' self-reported Examinations Committees with the endorsement of the Executive Committee of the ESICM decided to perform a series of measures in order to further improve the quality and standardization of the EDIC exam. This includes: (1) unification of the electronic database system to allow prospective candidates' data collection and regular exam outcome evaluation; (2) introducing the individual examiner's performance monitoring 12 ; (3) introducing a scheme of regular examiners' rotations among centres; (4) gaining accreditation of The Council for European Specialists Medical Assessment (CESMA) for the EDIC exam.…”

Section: Discussionmentioning

confidence: 99%

Factors associated with success in the oral part of the European Diploma in Intensive Care

Waldauf

Rubulotta

Sitzwohl

et al. 2017

Journal of the Intensive Care Society

View full text Add to dashboard Cite

Introduction: The oral part of European Diploma in Intensive Care diploma examinations changed in 2013 into an objective structured clinical examination-type exam. This step was undertaken to provide a fair and reproducible clinical exam. All candidates face identical questions with predefined correct answers simultaneously in seven high throughput exam centres on the same day. We describe the factors that are associated with success in part 2 European Diploma in Intensive Care exam. Methods: We prospectively collected self-reported data from all candidates sitting European Diploma in Intensive Care part 2 in 2015, namely demographics, professional background and attendance to a European Diploma in Intensive Care part 2 or generic objective structured clinical examination preparatory courses. After testing association with success (with cutoff at p < 0.10) and co-linearity of these factors as independent variables, we performed a multivariate logistical analysis, with binary exam outcome (pass/fail) as the dependent variable. Structural equation modelling was used to gain further insight into relations among determinants of success in the oral part of the European Diploma in Intensive Care. Results: Out of 427 candidates sitting the exam, completed data from 341 (80%) were available for analysis. The following candidates' factors were associated with increased chance of success: English as native language (odds ratio 4.3 (95% CI 1.7-10.7)), use of Patient-centred Acute Care Training e-learning programme module (odds ratios 2.0 (1.2-3.3)), working in an EU country (odds ratios 2.5 (1.5-4.3)), and better results in the written part of the European Diploma in Intensive Care (for each additional SD of 6.1 points odds ratios 1.9 (1.4-2.4)). Chance of success in the European Diploma in Intensive Care 2 decreased with increased candidates 'age (for each additional SD of 5.5 years odds ratios 0.67 (0.51-0.87)). Exam centres (7 in total) could be clustered into 3 groups with similar success rates. There were significant differences in exam outcomes among these 3 groups of exam centres even after adjustment to known candidates' factors (G1 vs G2 odds ratios 2.4 (1.4-4.1); G1 vs G3 odds ratios 9.7 (4.0-23.1) and G2 vs G3 odds ratios 3.9 (1.7-9.2)). A short data collection period (only one year) and 20% of missing candidates' data are the main limitations of this study. Conclusions: Younger age, English as native language, better results in written part of the exam, working at a European country and the use of PACT for preparation, were factors associated with success in the oral part of the European Diploma in Intensive Care exam. Despite the limitations of this study, the differences in outcome among the exam centres will need further investigation.

show abstract

A method for identifying extreme OSCE examiners

Cited by 29 publications

References 11 publications

“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

“On the same page”? The effect of GP examiner feedback on differences in rating severity in clinical assessments: a pre/post intervention study

Managing extremes of assessor judgment within the OSCE

Factors associated with success in the oral part of the European Diploma in Intensive Care

Contact Info

Product

Resources

About