Using Forest Plots to Present the Performance Sheets Provided to OSCE Examinees Using Rasch Analysis (Preprint)

Abstract

BACKGROUND Objective structured clinical examination (OSCE) has become an acceptable method for evaluating clinical skills in healthcare settings. However, none of the current literature has used visualization plots to compare individual OSCE performances. OBJECTIVE This study uses the Rasch model to (1) compare two other scenarios composed of binary scores and mixed binary and percentage responses; (2) draw a forest plot to visualize students’ performance; and (3) observe the comparison of OCSE scores among subgroups (e.g., schools) using real data.. METHODS We obtained data from 29 medical students who participated in OSCE examinations at 12 stations in 2018. The continuous Rasch model (CRM) was applied to estimate person measures and standard errors (SEs). The characteristics of the percentage CRM were interpreted in comparison to the other two counterparts (i.e., binary and mixed scenarios). Forest plots frequently used in meta-analysis were drawn to display those 29 examinees and five subgroups. RESULTS We used online CRM to draw forest plots and observed that (1) the binary scenario had a wider range of measures, followed by the mixed scenario. The percentage mode as expected has narrower ranges in person abilities(from -0.42 to 0.38) and item difficulties(from -0.48 to 0.41) owing to smaller total raw scores when compared to the other two counterparts;(2) all OSCE performances of medical students do not deviate from the middle line with zero logits; (3) The students in Taipei Medical University outperformed other schools among the five subgroups. CONCLUSIONS We designed two online computer modules for (1) estimating person measures and SEs and (2) drawing forest plots to display the OSCE performance sheets for Taiwanese medical students, which helps hospitals evaluate OSCE performances for students efficiently and objectively, but such results are not limited to medical students in the future.