Scoring Objective Structured Clinical Examinations Using Video Monitors or Video Recordings

Abstract

Objective. To compare scoring methods for objective structured clinical examinations (OSCEs) using real-time observations via video monitors and observation of videotapes. Methods. Second-(P2) and third-year (P3) doctor of pharmacy (PharmD) students completed 3-station OSCEs. Sixty encounters, 30 from each PharmD class, were selected at random, and scored by faculty investigators observing video monitors in real-time. One month later, the encounters were scored by investigators using videotapes. Results. Intra-rater reliability between real-time and videotaped observation was excellent (ICC 3,1 of 0.951 for P2 students and 0.868 for P3 students). However, 13.3% of students' performance in both P2 and P3 cohorts changed in pass/fail determination from passing based on real-time observation to failing based on video observation, and 3.3% of students changed from failing real-time to passing on video. Conclusions. Despite excellent overall reliability, important differences in OSCE pass/fail determinations were found between real-time and video observations. These observation methods for scoring OSCEs are not interchangeable.

Effects of a rater training on rating accuracy in a physical examination skills assessment

“…The authors point out that the range was similar to previously published interrater-reliability scores of live raters [ 23 ]. A second study with pharmacy students specifically studied the intra-rater reliability after one month [ 24 ]. The reliability was high; however, due to a higher stringency in the video rating, more candidates would have failed in the post-hoc assessment.…”

Training for Failure: A Simulation Program for Emergency Medicine Residents to Improve Communication Skills in Service Recovery

“…Second, the time and method of observation differed between the faculty evaluators and the PIs, which may have inadvertently impacted the scores. Other studies have found real‐time and video observations may not be interchangeable . The PIs scored the students in the 5 minutes after the case, whereas the faculty observers had as much time as they needed to view the videos and evaluate the performance.…”

Educational Testing and Validity of Conclusions in the Scholarship of Teaching and Learning

“…In the pharmacy education literature, there is an example of using an ICC to determine the degree of agreement between the analytical checklist scores obtained in 2 different conditions (real-time and video). 24 The ICC was 0.951, which the authors interpreted as high agreement (values of less than 0.4 indicated poor agreement; between 0.4 and 0.8, fair to good agreement; and greater than 0.8, excellent agreement). An example of Cohen kappa can be found in an analysis of letter grades from the 2008-2009 academic year using 2 independent faculty evaluations representing categorical-level data.…”

“…The researchers reported high rubric reliability (r50.98) and recommended that several grades be adjusted to eliminate evaluator leniency, although they concluded that evaluator leniency appeared minimal. 24 …”

“…Reproaching the example from kappa above, that article is also an example of a Rasch analysis of a PharmD student presentation rubric and judge bias. 24 In that study, a Many-Facets Rasch model was used "to determine the rubric's reliability, quantify the contribution of evaluator harshness/leniency in scoring, and assess grading validity by comparing the current grading method with a criterion-referenced grading scheme." The researchers reported high rubric reliability (r50.98) and recommended that several grades be adjusted to eliminate evaluator leniency, although they concluded that evaluator leniency appeared minimal.…”