“EMERGing” Electronic Health Record Data Metrics: Insights and Implications for Assessing Residents’ Clinical Performance in Emergency Medicine

Abstract

Objectives: Competency-based medical education requires that residents are provided with frequent opportunities to demonstrate competence as well as receive effective feedback about their clinical performance. To meet this goal, we investigated how data collected by the electronic health record (EHR) might be used to assess emergency medicine (EM) residents' independent and interdependent clinical performance and how such information could be represented in an EM resident report card.Methods: Following constructivist grounded theory methodology, individual semistructured interviews were conducted in 2017 with 10 EM faculty and 11 EM residents across all 5 postgraduate years. In addition to open-ended questions, participants were presented with an emerging list of EM practice metrics and asked to comment on how valuable each would be in assessing resident performance. Additionally, we asked participants the extent to which each metric captured independent or interdependent performance. Data collection and analysis were iterative; analysis employed constant comparative inductive methods.Results: Participants refined and eliminated metrics as well as added new metrics specific to the assessment of EM residents (e.g., time between signup and first orders). These clinical practice metrics based on data from our EHR database were organized along a spectrum of independent/interdependent performance. We conclude with discussions about the relationship among these metrics, issues in interpretation, and implications of using EHR for assessment purposes.Conclusions: Our findings document a systematic approach for developing EM resident assessments, based on EHR data, which incorporate the perspectives of both clinical faculty and residents. Our work has important implications for capturing residents' contributions to clinical performances and distinguishing between independent and interdependent metrics in collaborative workplace-based settings.

Using Resident-Sensitive Quality Measures Derived From Electronic Health Record Data to Assess Residents’ Performance in Pediatric Emergency Medicine

“…It could also be due to greater interdependence between faculty and first-year residents, in which less experienced residents are more likely to review orders with their faculty before placing them in the EHR. 35 As residents advance in their training and gain more independence, they are also given more responsibility and harder tasks, leading to a lower scores because the task (i.e., the requisite patient care) is harder. 36 In this study, the apparent overrepresentation of first-year residents in the top tertile seems to be accounted for by seeing fewer patients overall, seeing fewer acute patients, or potentially spending more time on patient care.…”

“…Further research could also focus on studying cross-classification effects stemming from the interdependent nature of team-based patient care 39 while developing the methodological ingenuity needed to capture both nesting and cross-effects when measuring such performance. 35 , 40 …”

Replacing high-stakes summative examinations with graduated medical licensure in Canada

“…A modern model of assessment for licensure that integrates both training programs' competency-based assessments and clinical practice data derived from trainees' clinical work would better align with evidence-informed educational practices, underscore the importance of continuous quality improvement for practitioners and refocus licensure on realworld clinical practice. 10,11 The implementation of such a system would be challenging. Most international medical training programs use high-stakes, gatekeeping examinations similar to those used in Canada, so cannot provide robust examples to guide the way forward.…”

Resident-sensitive quality measures

“…65 Paradoxically, quality measures sometimes attributed to supervising physicians may actually be attributable to residents. 66 Once quality measures relevant to the work that residents perform are clearly delineated, scholars have advocated the use of electronic health records (EHRs) to produce such measures. 29 The EHR has already been used to determine resident experience (e.g., conditions seen), 67 making its use for obtaining quality metrics for residents a logical next step.…”

“…Sebok-Syer and colleagues are currently beginning to explore this area, seeking to determine the spectrum of interdependence in emergency medicine performance metrics for residents. 66 For example, they have found that the metric of "turnaround time between signup [for a patient] and first orders" is felt to be 94% attributable to the resident, ordering of lab tests and medications are at or near 50%, time to fluids is only 16% attributable to the resident, and length of stay is 0% attributable to the resident. Expanding this work in the future will be important not only for further understanding the interdependence of care, an interest of Sebok-Syer and colleagues, 67 but also for further understanding and developing RSQMs to determine which quality metrics are, and are not, largely attributable to individual residents.…”