Anchor-based, end-of-shift ratings are commonly used to conduct performance assessments of resident physicians. These performance evaluations often include narrative assessments, such as solicited or “free-text” commentary. Although narrative commentary can help to create a more detailed and specific assessment of performance, there are limited data describing the effects of narrative commentary on the global assessment process. This single-group, observational study examined the effect of narrative comments on global performance assessments. A subgroup of the clinical competency committee, blinded to resident identity, assigned a single, consensus-based performance score (1–6) to each resident based solely on end-of-shift milestone scores. De-identified narrative comments from end-of-shift evaluations were then included and the process was repeated. We compared milestone-only scores to milestone plus narrative commentary scores using a nonparametric sign test. During the study period, 953 end-of-shift evaluations were submitted on 41 residents. Of these, 535 evaluations included free-text narrative comments. In 17 of the 41 observations, performance scores changed after the addition of narrative comments. In two cases, scores decreased with the addition of free-text commentary. In 15 cases, scores increased. The frequency of net positive change was significant ( p = .0023). The addition of narrative commentary to anchor-based ratings significantly influenced the global performance assessment of Emergency Medicine residents by a committee of educators. Descriptive commentary collected at the end of shift may inform more meaningful appraisal of a resident’s progress in a milestone-based paradigm. The authors recommend clinical training programs collect unstructured narrative impressions of residents’ performance from supervising faculty.
Background
Accurate electrocardiographic (ECG) differentiation of ventricular tachycardia (VT) from supraventricular tachycardia with aberrancy (SVT-A) on ECG is key to therapeutic decision-making in the emergency department (ED) setting.
Objective
The goal of this study was to test the accuracy and agreement of emergency medicine residents to differentiate VT from SVT-A using the Vereckei criteria.
Methods
Six emergency medicine residents volunteered to participate in the review of 114 ECGs from 86 patients with a diagnosis of either VT or SVT-A based on an electrophysiology study. The resident reviewers initially read 12-lead ECGs blinded to clinical information, and then one week later reviewed a subset of the same 12-lead ECGs unblinded to clinical information.
Results
One reviewer was excluded for failing to follow study protocol and one reviewer was excluded for reviewing less than 50 blinded ECGs. The remaining four reviewers each read 114 common ECGs blinded to clinical data and their diagnostic accuracy for VT was 74% (sensitivity 70%, specificity 80%), 75% (sensitivity 76%, specificity 73%), 61% (sensitivity 81%, specificity 25%), and 68% (sensitivity 84%, specificity 40%). The intraclass correlation coefficient (ICC) was 0.31 (95% CI 0.22 – 0.42). Eliminating two of the four reviewers who left a disproportionately high number of ECGs unclassified resulted in an increase in overall mean diagnostic accuracy (70% to 74%) and agreement (0.31 to 0.50) in the two remaining reviewers. Three reviewers read 45 common ECGs unblinded to clinical information and had accuracies for VT 93%, 93% and 78%.
Conclusion
The new single lead Vereckei criteria, when applied by emergency medicine residents achieved only fair-to-good individual accuracy and moderate agreement. The addition of clinical information resulted in substantial improvement in test characteristics. Further improvements (accuracy and simplification) of algorithms for differentiating VT from SVT-A would be helpful prior to clinical implementation.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.