Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an Internal Medicine residency OSCE

Abstract

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.

Consensus-Based Expert Development of Critical Items for Direct Observation of Point-of-Care Ultrasound Skills

“…Critical items can help guide faculty development efforts by helping them better focus on key essential tasks, thereby more effectively managing rater workload 56 and improving rater performance. 57 Secondly, using key items in assessments may potentially result in higher diagnostic accuracy 30,51 and superior reliability measures, 58 training, and patient safety. 29 In the era of competency-based medical education, 11 mastery-based learning is associated with improved clinical outcomes.…”

Using standardized patients to evaluate medical students’ evidence‐based medicine skills

“…Furthermore, some students only used pre‐assessed sources such as Uptodate, which limited our ability to evaluate EBM appraisal skills. Although our checklist was not validated, it was reviewed by content experts to increase reliability and objectivity …”

“…Although our checklist was not validated, it was reviewed by content experts to increase reliability and objectivity. (14)(15)(16)…”

Passing a Technical Skills Examination in the First Year of Surgical Residency Can Predict Future Performance

“…However, while the focus was on surgery trainees, the results of this study could be of interest to a broader surgical audience, as the OSATS has been widely adopted across other surgical specialties 8,13,20,21 and anesthesiology. 22 Furthermore, the OSATS, as a performance-based assessment, parallels the objective structured clinical examination, 23 which assesses clinical skills and has been used in nontechnical specialties, including internal medicine 24,25 and family medicine. 26 While this study focused on surgery, it provides foundational work for further predictive studies in other technical and nontechnical specialties.…”