Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace: Findings From One Institution

Abstract

PurposeAssessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships.

A Question of Scale? Comparison of Generalizability in Ottawa and Chen Scales When Used to Formulate Ad Hoc Entrustment Decisions for the Core EPAs

“…The frequent assessor model seemed to increase variance attributed to the learner for the Chen scale in only 2 Core EPAs. Overall, these findings were similar to our previous study 4 involving only the Ottawa scale. Based on these findings in conjunction with prior evidence, we suggest that the root cause analysis for challenges associated with WBAs for the Core EPAs involves the lack of a true competency-based curriculum in UME; the choice of scale alone does not appear to impact performance.…”

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

“…A summary of the results following application of the frequent assessor model is found in Supplemental Digital Appendix 2, at http://links.lww.com/ ACADMED/B130. A comparison of the overall findings from this study with those of our previous study 34 is found in Table 3.…”

“…While that is of some concern, the moderate correlation observed between scores suggests that raters were able to select different levels of performance even when using both simultaneously. Third, in line with the experience from the Core EPA pilot and prior research, 28,29,34 we used modified versions of the Ottawa and Chen scales for our WBAs. It may be worth investigating whether the original scales perform better.…”

“…The WBA system we used to generate data for this study has been described previously in detail. 34,37 In brief, all medical students were required to request a specified number of direct observations using a mobile-friendly WBA instrument linked to the learning management system. Clerkship directors identified the Core EPAs most appropriate to observe in their respective clerkship.…”

“…Reasons for these challenges included the large number of raters, limited rater training, and issues related to the Ottawa scale itself when used in the UME context to assess performance of the Core EPAs. 34…”