Clinical Competency Assessments: A Comparative Study of Virtual-Reality-Based and Traditional Physical OSCE Stations (Preprint)

Abstract

BACKGROUND Objective structured clinical examinations (OSCEs) are a widely recognized and accepted method to assess clinical competencies but are often resource-intensive. Moreover, they may not comprehensively capture the complexity of emergency scenarios. OBJECTIVE This study aimed to evaluate the feasibility and effectiveness of a virtual reality (VR) station compared to traditional physical stations in an already established curricular OSCE. METHODS Fifth-year medical students participated in an OSCE that included ten stations in total, with one station dedicated to emergency medicine offered in two formats and featuring scenarios of septic and anaphylactic shock in each format. Participants in the study were randomly divided into two groups, participating either in the virtual-reality station (VRS) or the physical station (PHS). Student performance and item characteristics were analyzed focusing on the one emergency and the five other case-based stations; four technical-skills-oriented stations were excluded from this study. Student perceptions were recorded as part of a post-examination online survey to assess the acceptance and usability of VR. RESULTS Following randomization and exclusions of invalid datasets, 57 and 66 participants were assessed for the VRS and PHS, respectively. The two VRS scenarios (septic and anaphylactic shock) integrated well and demonstrated a balanced level of difficulty (P = 0.67 and 0.58, respectively) with an average difficulty of 0.68 across all stations. They exhibited above-average values with respect to item discrimination (r' = 0.40/0.33, overall = 0.30) and discrimination index (D = 0.25/0.26, overall = 0.16). VRS participant responses emphasized the realistic portrayal of medical emergencies and the fair assessment conditions provided. However, there was some hesitancy towards its broader application in future practical assessments, highlighting the need both for further familiarization as well as maintaining physical interaction with simulated patients. CONCLUSIONS Integration of the VRS into the current OSCE framework proved feasible both technically and organizationally, even within the strict constraints of short examination phases and schedules. The VRS was accepted and positively received by students across various levels of technological proficiency, including those with no prior VR experience. Notably, the VRS demonstrated comparable or even superior item characteristics, particularly in terms of discrimination power. While challenges remain, such as technical reliability and some acceptance concerns, VR remains promising in applications of clinical competence assessment.