An Event‐based Approach to Measurement: Facilitating Observational Measurement in Highly Variable Clinical Settings

Abstract

Background: Translational research in medical education requires the ability to rigorously measure learner performance in actual clinical settings; however, current measurement systems cannot accommodate the variability inherent in many patient care environments. This is especially problematic in emergency medicine, where patients represent a wide spectrum of severity for a single clinical presentation. Our objective is to describe and implement EBAM, an event-based approach to measurement that can be applied to actual emergency medicine clinical events. Methods:We used a four-step event-based approach to create an emergency department trauma resuscitation patient care measure. We applied the measure to a database of 360 actual trauma resuscitations recorded in a Level I trauma center using trained raters. A subset (n = 50) of videos was independently rated in duplicate to determine inter-rater reliability. Descriptive analyses were performed to describe characteristics of resuscitation events and Cohen's kappa was used to calculate reliability. Results:The methodology created a metric containing both universal items that are applied to all trauma resuscitation events and conditional items that only apply in certain situations. For clinical trauma events, injury severity scores ranged from 1 to 75 with a mean (AESD) of 21 (AE15) and included both blunt (254/360; 74%) and penetrating (86/360; 25%) traumatic injuries, demonstrating the diverse nature of the clinical encounters. The mean (AESD) Cohen's kappa for patient care items was 0.7 (AE0.3).

Does team leader gender matter? A Bayesian reconciliation of leadership and patient care during trauma resuscitations

“…The team leadership and patient care measures were developed through an iterative process that provided evidence of validity as described by Cook et al 15 The specific steps included a thorough review of the literature and input from subject experts (content validity), as well as rater training and duplicate coding of a portion of the observations to determine interrater reliability (internal structure) across a range of performance levels. 5,16 Additionally, scores from the team leadership measure were shown to correlate with patient care measure scores as predicted by conceptual models of functional team leadership (relationship to other variables). 5,17 The leadership measure targeted important team leadership behaviors identified through 2 systematic reviews and subject matter expert input.…”

“…The measure, as well as additional information regarding the supporting validity evidence, was published with the original study. 16 Two independent groups of trained raters coded the observations for team leadership (n = 4) and patient care (n = 2). Raters were balanced with regard to gender and blinded to the study hypotheses.…”