Item Analysis to Improve Reliability for an Internal Medicine Undergraduate OSCE

Abstract

Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ''problem stations'' were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.

Implementation of Objective Structured Clinical Examination for Assessing Nursing Students’ Clinical Competencies: Lessons and Implications

“…It is also accepted as a fitfor-purpose instrument for measuring clinical reasoning skills (Ahmed, 2009). When compared to traditional methods of clinical skills assessment, OSCE has several advantages, and it meets the two cardinal criteria of an effective assessment tool vis-à-vis validity and reliability (Gormely, 2011;Turner & Dankoski, 2008;Auewarakul et al, 2005).…”

Objective Structured Clinical Examination as an assessment method for undergraduate medical students

“…At the end of the semester, each student took the written exam, which was analyzed (mean, median, mode) separately for each year. [21][22][23][24][25][26][27] The validity was measured using Pearson correlation. Each system was analyzed to understand any deficits in the courses.…”

“…The R 2 coefficient is used to measure the proportional change in the dependent variable (the checklist score) by changes in the independent variable (the global grade); this is a marker of internal consistency. [21][22][23][24][25][26] The final method for establishing the validity of the exam is the comparison of the OSCE scores with written exam scores using Pearson correlation. [27] In Saudi Arabia, King Saud University and King Abdulaziz University were the first Saudi universities to published their experiences about using the OSCE in the field of surgery, which was reported in detail by Al-Naami [28] at King Saud University (for students in their final year).…”

American College of Surgeons Resident Objective Structured Clinical Examination

“…Reliability can generally be improved by increasing the number of items and/or stations with consequent increase in the duration of an examination and using item analysis to identify those that "perform well" with respect to their correlation with other items, stations and/or overall scores. 19,20 These issues were considered in developing the ACS OSCE, and only the essential items were retained on the checklists before launching the pilot study. A post hoc analysis was also conducted to review the stations and the associated checklists meticulously to ensure that they are consistent with current evidence-based practices.…”