Workplace-Based Entrustment Scales for the Core EPAs: A Multisite Comparison of Validity Evidence for Two Proposed Instruments Using Structured Vignettes and Trained Raters

Abstract

PurposeIn undergraduate medical education (UME), competency-based medical education has been operationalized through the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). Direct observation in the workplace using rigorous, valid, reliable measures is required to inform summative decisions about graduates' readiness for residency. The purpose of this study is to investigate the validity evidence of 2 proposed workplace-based entrustment scales. MethodThe authors of this multisite, randomized, experimental study used structured vignettes and experienced raters to examine validity evidence of the Ottawa scale and the UME supervisory tool (Chen scale) in 2019.The authors used a series of 8 cases (6 developed de novo) depicting learners at preentrustable (less-developed) and entrustable (more-developed) skill levels across 5 Core EPAs. Participants from Core EPA pilot institutions rated learner performance using either the Ottawa or Chen scale. The authors used descriptive statistics and analysis of variance to examine data trends and compare ratings, conducted interrater reliability and generalizability studies to evaluate consistency among participants, and performed a content analysis of narrative comments. ResultsFifty clinician-educators from 10 institutions participated, yielding 579 discrete EPA assessments. Both Ottawa and Chen scales differentiated between less-and more-developed skill levels (P < .001). The interclass correlation was good to excellent for all EPAs using Ottawa (range, 0.68-0.91) and fair to excellent using Chen (range, 0.54-0.83). Generalizability analysis revealed substantial variance in ratings attributable to the learner-EPA interaction (59.6% for Ottawa; 48.9% for Chen) suggesting variability for ratings was appropriately associated with performance on individual EPAs.

Comparing Entrustment Decision-Making Outcomes of the Core Entrustable Professional Activities Pilot, 2019-2020

“…Trained faculty were convened to make determinations about readiness for indirect supervision looking at multimodal performance data, including Workplace Based Assessments (WBAs) with entrustment-supervision scales . These determinations were generally completed within a day after direct observation in the workplace …”

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

“…33 While both of these studies provide some evidence, they were limited by focus on a limited number of senior-year students 29 or implementation in a standardized environment. 33 In the 2018-2019 academic year, we implemented WBAs that incorporated the Ottawa scale across clerkships at the Virginia Commonwealth University School of Medicine (VCU-SOM). We then conducted generalizability studies (G-studies) to determine the internal structure validity evidence of the scale.…”

“…Reasons for these challenges included the large number of raters, limited rater training, and issues related to the Ottawa scale itself when used in the UME context to assess performance of the Core EPAs. 34 In response to our prior findings of low reliability/generalizability when implementing the Ottawa scale alone 34 and the limitations of prior studies assessing both the Ottawa and Chen scales in the UME setting 29,33 we incorporated both the Ottawa and Chen scales into our WBAs for the 2019-2020 academic year. Our primary purpose with this study was to compare performance, in terms of validity and reliability data, of the 2 scales head-to-head when used to assess a large cohort of students observed during clerkships throughout the academic year.…”

The “Growth Instant Feedback Tool (GIFTool)” Workplace-Based Assessment of Entrustable Professional Activity Performance: Preliminary Evidence for Validity

“…In a small, single center study comparing the reliability and generalizability of the OCAT and Chen scales in assessing clerkship students, both scales performed relatively poorly in terms of variance related to the learner (9). However, when faculty from multiple core EPA schools used the two scales to assess a learner in video vignettes, both scales showed similar reliability, similar variance related to students, and the OCAT had good to excellent inter-rater reliability for the majority of EPAs evaluated (10). In a single site study comparing the two scales, prospective ratings of students (Chen Scale) tended to be higher than retrospective ratings (OCAT) and rater preference for the two scales varied based on setting and clinical activity (14).…”

“…In addition, the OCAT and Chen scales are limited by the fact that faculty may vary in their own beliefs regarding how much supervision should be provided to trainees (10). This may be particularly true of outpatient faculty who have limited interactions with medical students.…”