How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49

Abstract

Practice points It is important to always evaluate the quality of a high stakes assessment, such as an OSCE, through the use of a range of appropriate metrics. When judging the quality of an OSCE, it is very important to employ more than one metric to gain an all round view of the assessment quality. AbstractWith an increasing use of criterion based assessment techniques in both undergraduate and postgraduate healthcare programmes, there is a consequent need to ensure the quality and rigour of these assessments. The obvious question for those responsible for delivering assessment is how is this 'quality' measured, and what mechanisms might there be that allow improvements in assessment quality over time to be demonstrated? Whilst a small base of literature exists, few papers give more than one or two metrics as measures of quality in OSCEs.In this guide, aimed at assessment practitioners, the authors aim to review the metrics that are available for measuring quality and indicate how a rounded picture of OSCE assessment quality may be constructed by using a variety of such measures, and also to consider which characteristics of the OSCE are appropriately judged by which measure(s). The authors will discuss the quality issues both at the individual station level and across the complete clinical assessment as a whole, using a series of 'worked examples' drawn from OSCE data sets from the authors' institution.

Impact of Task-based Checklist Scoring and Two Domains Global Rating Scale in Objective Structured Clinical Examination of Pharmacy Students.

“…The reason for unsatisfactory association between checklist marks and global ratings is,mathematically a cubic will always produce a better fit, but stingily dictates that the dissimilarity among the two fits has to be statistically substantial for a higher order model to be preferred. 11 The reason for the fit of the cubic polynomial is significantly better in station 7 is better explained in a previous study 12 which states that, the key point to note is whether the cubic expression is the result of an underlying relationship or as a result of outliers, resulting from inappropriate checklist design or unacceptable assessor behavior in marking.…”

Using a ‘Yellow Card’ in the Objective Structured Clinical Exam: Does it Add to the Identification of Problem Postgraduate Trainees in General Practice? An Exploratory Study to Identify High Risk Trainees

“…First, the validity of the OSCE in assessing clinical competencies needs to be questioned. Debate on the added value of an OSCE as compared to oral exams (from jury exams to the so called long case) remains active [7,16,17]. Opponents of the OSCE dispute the content reliability and veracity.…”

“…Therefore item checklists are developed as formal tools to score the performance of the individual trainee in each station. The final score is then calculated from the score on the item checklist, a separate appraisal on general vocational skills and a global score given by the observer [7].…”

“…The OSCE is considered as a reliable and objective tool to evaluate clinical competences in standardized patient encounters [4]. Although this test is generally accepted as a high stake assessment process, reliability, validity and reproducibility of an OSCE remains subject of discussions and concerns [5][6][7]. On trainee level, there is an ongoing debate on the efficacy of evaluating skills using an item-checklist [5].…”

Academic performance of fnal year medical students at kerbala medical college

“…OSCE assess history taking, clinical examination, data interpretation procedure or practical skills and communication skills. [11][12][13] It represents a valid and reliable tool in medical education for evaluating clinical competence. 13 OSCE was introduced to test final year candidates in the Council of Arab Board students and later for undergraduate students in Iraq.…”