Implementing a Real-time Workplace-based Assessment Data Collection System Across an Entire Medical School’s Clinical Learning Environment

Abstract

Background & Objectives: Workplace-based assessments (WBAs) are a vital aspect of medical student competency assessment for the core Entrustable Professional Activities (EPAs), but pose significant challenges since assessment must occur in real-time during the routine care of patients. We developed an online WBA system designed to overcome these challenges, and implemented it across an entire undergraduate medical education program to address the need for EPA competency assessment. We describe the development and implementation process, and present initial results from our inaugural medical student cohort. Methods:The WBA tool was designed to be student-driven, easy to use, and minimally disruptive to clinical care. Students trigger assessments by choosing the desired EPA to be assessed within a custom-built Qualtrics XM survey application. Their clinical assessor is prompted to select their level of involvement in the activity using the modified Ottawa co-activity scale and provide brief written feedback. Direct verbal feedback at time of discussion is encouraged.Results: 3,568 WBAs were completed. The mean number of assessments per student for all EPAs combined was 24.27 with a range of 1-103. All students completed at least one WBA. Over the course of 12 months, the mean number of EPAs recorded per student in this cohort was lowest for EPA 10 (Recognizing a Patient Requiring Urgent or Emergent Care and Initiate Evaluation and Management) (mean=0.36; range 0-4; n=53) and was highest for EPA 6 (Provide an Oral Presentation for a Clinical Encounter) (mean=5.46; range 1-17; n=803). The mean number of minutes it took to complete the assessments was 2.7 minutes with a standard deviation of 1.2 minutes (n=2,803).

An Analysis of Workplace-Based Assessments for Core Entrustable Professional Activities for Entering Residency: Does Type of Clinical Assessor Influence Level of Supervision Ratings?

“…Finally, advanced practice providers, nurses, and physician assistants complete fewer WBAs overall and usually only for EPA 9. All these factors may affect students’ ratings 7–28 …”

“…A health system–wide online WBA data collection system was launched across the entire undergraduate medical education (UME) program to visualize a learner’s evolving competency in a variety of authentic clinical settings over time to foster EPA assessment. Full details about the creation of the WBA platform can be found in the article by Hasan et al 19 To graduate, all learners must receive a requisite number of WBAs across the 13 EPAs. Students can initiate WBAs at any point during their clinical rotations by requesting a supervisor complete an online assessment.…”

Assessment in Undergraduate Competency-Based Medical Education: A Systematic Review

“…With this, they may use it to assume more responsibility for their own learning. By providing immediate feedback, a VLE may generally improve formative assessment [ 70 ].…”

Leveraging existing education innovations to establish a community of practice to promote medical education scholar development

“…Fed by multiple data streams, such as ExamSoft, MedHub, Anatomy Hive and USMLE/NBME, the R esearch & E valuation D ata for E ducational I mprovement (REDEI) system houses student performance on a variety of formative and summative educational metrics. REDEI’s dashboards include performance on weekly quizzes, end of block examinations (e.g., OHSU final exams, clinical skills and performance exams, USMLE/NBME assessments, and health systems science exams) [ 15 ], as well as workplace-based assessments (WBAs) aligned with the 13 Core Entrustable Professional Activities (EPAs) for entering residency [ 16 ]. Scholarly projects, student artifacts, preceptorship, and remediation data are also included in REDEI.…”