Feedback Frequency in Competence by Design: A Quality Improvement Initiative

Abstract

Background Otolaryngology–head and neck surgery is in the first wave of residency training programs in Canada to adopt Competence by Design (CBD), a model of competency-based medical education. CBD is built on frequent, low-stakes assessments and requires an increase in the number of feedback interactions. The University of Toronto otolaryngology–head and neck surgery residents piloted the CBD model but were completing only 1 assessment every 4 weeks, which was insufficient to support CBD. Objective This project aimed to increase assessment completion to once per resident per week using quality improvement methodology. Methods Stakeholder engagement activities had residents and faculty characterize barriers to assessment completion. Brief electronic assessment forms were completed by faculty on residents' personal mobile devices in face-to-face encounters, and the number completed per resident was tracked for 10 months during the 2016–2017 pilot year. Response to the intervention was analyzed using statistical process control charts. Results The first bundled intervention—a rule set dictating which clinical instance should be assessed, combined with a weekly reminder implemented for 10 weeks—was unsuccessful in increasing the frequency of assessments. The second intervention was a leaderboard, designed on an audit-and-feedback system, which sent weekly comparison e-mails of each resident's completion rate to all residents and the program director. The leaderboard demonstrated significant improvement from baseline over 10 weeks, increasing the assessment completion rate from 0.22 to 2.87 assessments per resident per week. Conclusions A resident-designed audit-and-feedback leaderboard system improved the frequency of CBD assessment completion.

Graduate medical education-led continuous assessment of burnout and learning environments to improve residents’ wellbeing

“…19 Use of PDSA to improve medical education has been reported. Arnstead et al used PDSA cycles to improve the frequency of Competence by Design assessment in surgery residents in Canada [ 26 ]. Dunbar et al demonstrated that using PDSA cycles improved physician recognition and reporting of patient safety events [ 27 ],…”

Application of continuous quality improvement to medical education

“…PDSA has the advantage of building knowledge quickly through the efficient use of data and relying on a high 'return on failure ratio' 33 Explicit use of PDSA to improve medical education has been reported in several different contexts, including efforts to improve safety event reporting 36 or the number of resident assessments by faculty members. 37 There are also other examples where PDSAlike approaches have been used for programme development and curriculum adaptation in medical education. For example, a faculty members development programme for education scholarship at the University of Toronto used a deliberate strategy to allow programme directors to receive inputs from multiple sources to inform iterative, real-time improvements to the programme design and delivery.…”

“…Explicit use of PDSA to improve medical education has been reported in several different contexts, including efforts to improve safety event reporting 36 or the number of resident assessments by faculty members 37 . There are also other examples where PDSA‐like approaches have been used for programme development and curriculum adaptation in medical education.…”