eOSCE stations live versus remote evaluation and scores variability

Abstract

Background Objective structured clinical examinations (OSCEs) are known to be a fair evaluation method. These recent years, the use of online OSCEs (eOSCEs) has spread. This study aimed to compare remote versus live evaluation and assess the factors associated with score variability during eOSCEs. Methods We conducted large-scale eOSCEs at the medical school of the Université de Paris Cité in June 2021 and recorded all the students’ performances, allowing a second evaluation. To assess the agreement in our context of multiple raters and students, we fitted a linear mixed model with student and rater as random effects and the score as an explained variable. Results One hundred seventy observations were analyzed for the first station after quality control. We retained 192 and 110 observations for the statistical analysis of the two other stations. The median score and interquartile range were 60 out of 100 (IQR 50–70), 60 out of 100 (IQR 54–70), and 53 out of 100 (IQR 45–62) for the three stations. The score variance proportions explained by the rater (ICC rater) were 23.0, 16.8, and 32.8%, respectively. Of the 31 raters, 18 (58%) were male. Scores did not differ significantly according to the gender of the rater (p = 0.96, 0.10, and 0.26, respectively). The two evaluations showed no systematic difference in scores (p = 0.92, 0.053, and 0.38, respectively). Conclusion Our study suggests that remote evaluation is as reliable as live evaluation for eOSCEs.

Feasibility and reliability of the pandemic-adapted online-onsite hybrid graduation OSCE in Japan

“…Online OSCE has an advantage in providing readily accessible video recordings for additional raters. Our results showed that assessments of online raters were not largely different from those of onsite raters, although online raters scored a little lower than onsite raters in 2021 hybrid OSCE, supporting previous data (Chan et al, 2014;Chen et al, 2019;Bouzid et al, 2022). Thus, the assessment of additional raters using recorded videos for only students who fell within BZ could be a reasonable solution without reducing feasibility.…”

Clues for improvement of research in objective structured clinical examination

“…Although our study does not allow establishing a causal link between these two observations, it is plausible that the pandemic may have prompted the development of research in this field. However, the data comparing remote/virtual to live OSCE are scarce, and only one recent study reported no significant difference between these two OSCE modalities [ 32 ]. These observations call for additional studies assessing the feasibility and reliability of remote-OSCE.…”

The Objective Structured Clinical Examination (OSCE) in Periodontology with Simulated Patient: The Most Realistic Approach to Clinical Practice in Dentistry

“…By referring to the examiners, the literature shows us that the teachers, and not the simulated patients, conducted the follow-up on the students [24]. In some cases, a highly protocolized calibrations is carried out.…”