Predictors of faculty narrative evaluation quality in medical school clerkships

Abstract

Introduction: Narrative approaches to assessment provide meaningful and valid representations of trainee performance. Yet, narratives are frequently perceived as vague, nonspecific and low quality. To date, there is little research examining factors associated with narrative evaluation quality, particularly in undergraduate medical education. The purpose of this study was to examine associations of faculty-and student-level characteristics with the quality of faculty member's narrative evaluations of clerkship students. Methods:The authors reviewed faculty narrative evaluations of 50 students' clinical performance in their inpatient medicine and neurology clerkships, resulting in 165 and 87 unique evaluations in the respective clerkships. The authors evaluated narrative quality using the Narrative Evaluation Quality Instrument (NEQI). The authors used linear mixed effects modelling to predict total NEQI score. Explanatory covariates included the following: time to evaluation completion, number of weeks spent with student, faculty total weeks on service per year, total faculty years in clinical education, student gender, faculty gender, and an interaction term between student and faculty gender.Results: Significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion, with NEQI scores decreasing by approximately 0.3 points every 10 days following students' rotations (p = .004).Additionally, women faculty had statistically higher quality narrative evaluations with NEQI scores 1.92 points greater than men faculty (p = .012). All other covariates were not significant. Conclusions:The quality of faculty members' narrative evaluations of medical students was associated with time to evaluation completion and faculty gender but not faculty experience in clinical education, faculty weeks on service, or the amount of time spent with students. Findings advance understanding on ways to improve the quality of narrative evaluations which are imperative given assessment models that will increase the volume and reliance on narratives.

Novel In-Training Evaluation Report in an Internal Medicine Residency Program: Improving the Quality of the Narrative Assessment

“…Without residents being physically present and with preceptor-resident interactions less retrievable from memory (sometimes weeks later), narrative assessments were less likely to be made. Interestingly, a 2022 follow-up study from the original developers of the NEQI found that “significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion.” 10 Consequently, the incorporation of learner goals in combination with the logistical changes to the evaluation process resulted in both more and higher quality narrative assessments.…”

“…Interestingly, a 2022 follow-up study from the original developers of the NEQI found that "significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion." 10 Consequently, the incorporation of learner goals in combination with the logistical changes to the evaluation process resulted in both more and higher quality narrative assessments.…”

A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

“…Secondly, we did not consider individual differences in student's, teachers' and clinical mentors' characteristics. Mooney et al [50], for example, found that feedback quality was associated with the feedback provider's gender. Additionally, a within-subject variation is possible, depending on the feedback performance phase being focused upon by the same feedback provider.…”

Automating the Identification of Feedback Quality Criteria and the CanMEDS Roles in Written Feedback Comments Using Natural Language Processing

“…First, the researchers did not collect demographic data about feedback providers. Previous research found that feedback quality could be associated with the gender of the feedback provider [47]. This may have caused bias in feedback comments and subsequently in the NLP model output.…”