A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments

Kelleher, Matthew; Kinnear, Benjamin; Sall, Dana; Schumacher, Daniel; Schauer, Daniel P.; Warm, Eric J.; Kelcey, Ben

doi:10.1097/acm.0000000000002997

Cited by 25 publications

(32 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Competency-based medical education (CBME) has brought increased focus on learner abilities, often measured with WBAs. While this method has the advantage of assessing real-world performance, there are several drawbacks including assessors using variable frames of reference 29 leading to suboptimal reliability for high-stakes decisions, [36][37][38] as well as the challenge of ensuring direct learner observation. 39 By reducing reliance on raters, RSQMs greatly reduce challenges of rater bias and supplement direct observation to give insight into resident performance.…”

Section: Discussionmentioning

confidence: 99%

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine

et al. 2020

Self Cite

View full text Add to dashboard Cite

BACKGROUND: Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM). OBJECTIVE: To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods. DESIGN: The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward. PARTICIPANTS: NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members. MAIN MEASURES: The number and description of RSQMs generated during this process. KEY RESULTS: Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM g e n e r a l c a r e -2 1 , d i a b e t e s m e l l i t u s -1 6 , hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission. CONCLUSIONS: Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…Dashboards to display these data are set to change color when a z-score is greater than one standard deviation below expected. We have been using this statistical modeling, a type of learning analytics, for several years to account for construct-irrelevant variance and help make sense of educational data [ 26 , 31 ].…”

Section: Methodsmentioning

confidence: 99%

“…The average for the control population was 3795 and 3715 for the study population. We chose the cumulative z‑score to identify our study population because previous findings in our program of assessment have shown higher reliability and it lessened the risk of including residents with temporary struggles [ 26 ]. Twenty-six of these residents were greater than one standard deviation below the mean z‑score (38% female).…”

Section: Methodsmentioning

confidence: 99%

“…These approaches highlight opportunities to better understand how narrative feedback analyzed early in training may differ between residents who struggle from those who do not struggle, while also describing the actual differences in the content of narrative data. In this study, we aimed to explore the first 6 months of narrative comments from WPBAs in one internal medicine residency program, with a robust program of assessment, [26][27][28][29] to determine identifiable patterns that subsequently might predict who will receive lower quantitative entrustment ratings over the course of training. Recognizing the early signal that portends the need for additional support and intervention at the beginning of residency can provide a practical approach for clinical competency committees (CCCs) to surface what matters from the plethora of positive, nonspecific, and or idiosyncratic narrative feedback.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program

et al. 2021

Self Cite

View full text Add to dashboard Cite

Introduction Narrative assessment data are valuable in understanding struggles in resident performance. However, it remains unknown which themes in narrative data that occur early in training may indicate a higher likelihood of struggles later in training, allowing programs to intervene sooner. Methods Using learning analytics, we identified 26 internal medicine residents in three cohorts that were below expected entrustment during training. We compiled all narrative data in the first 6 months of training for these residents as well as 13 typically performing residents for comparison. Narrative data were blinded for all 39 residents during initial phases of an inductive thematic analysis for initial coding. Results Many similarities were identified between the two cohorts. Codes that differed between typical and lower entrusted residents were grouped into two types of themes: three explicit/manifest and three implicit/latent with six total themes. The explicit/manifest themes focused on specific aspects of resident performance with assessors describing 1) Gaps in attention to detail, 2) Communication deficits with patients, and 3) Difficulty recognizing the “big picture” in patient care. Three implicit/latent themes, focused on how narrative data were written, were also identified: 1) Feedback described as a deficiency rather than an opportunity to improve, 2) Normative comparisons to identify a resident as being behind their peers, and 3) Warning of possible risk to patient care. Discussion Clinical competency committees (CCCs) usually rely on accumulated data and trends. Using the themes in this paper while reviewing narrative comments may help CCCs with earlier recognition and better allocation of resources to support residents’ development.

show abstract

“…34 OPAs As a Possible Solution for Assessing Noncognitive Abilities One process with emerging validity evidence for synthesizing disparate assessment elements into meaningful information is the observable practice activity (OPA) system. [35][36][37][38][39] OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, EPAs, or competencies, to create learning trajectories for formative and summative decisions. 35,36 OPAs drill down to observable behaviors that inform the whole and are similar conceptually to nested EPAs.…”

Section: Cbme and Observable Behaviorsmentioning

confidence: 99%

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills

et al. 2021

Self Cite

View full text Add to dashboard Cite

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients.

show abstract

A Reliability Analysis of Entrustment-Derived Workplace-Based Assessments

Cited by 25 publications

References 52 publications

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills

Contact Info

Product

Resources

About