Comparing the Quality of Narrative Comments by Rotation Setting

Roshan, Aishwarya; Wagner, Natalie; Acai, Anita; Emmerton‐Coughlin, Heather; Sonnadara, Ranil; Scott, Tracy; Karimuddin, Ahmer

doi:10.1016/j.jsurg.2021.06.012

Cited by 4 publications

(5 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the elaboration criterion was often lacking, which implies that crucial information making the feedback comments specific, was mostly missing. The paucity of specific feedback comments is a well-known phenomenon in healthcare education [14,23,31,32] and raises concerns about feedback providers' readiness and skills to provide feedback. While acknowledging their responsibility to provide students with high-quality feedback to foster competency development and performance improvement, feedback providers in healthcare education face considerable challenges in meeting this demand [33,34].…”

Section: Discussionmentioning

confidence: 99%

“…Training initiatives are available to train teachers and clinical mentors in writing high-quality feedback comments. Research about generic training initiatives -such as formal training programs and workshops -mirror inconsistent effects [31,45], mainly due to competing professional interests for time, including patient care, teaching, research, faculty promotion and administration [32]. As an alternative, a personalised approach could be adopted that focuses on giving feedback to feedback providers on their feedback comments and on coaching over a longer period of time [31].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

Van Ostaeyen,

Embo,

Rotsaert

et al. 2023

Perspect Med Educ

View full text Add to dashboard Cite

Introduction: Competency-based education requires high-quality feedback to guide students’ acquisition of competencies. Sound assessment and feedback systems, such as ePortfolios, are needed to facilitate seeking and giving feedback during clinical placements. However, it is unclear whether the written feedback comments in ePortfolios are of high quality and aligned with the current competency focus. Therefore, this study investigates the quality of written feedback comments in ePortfolios of healthcare students, as well as how these feedback comments align with the CanMEDS roles. Methods: A qualitative textual analysis was conducted. 2,349 written feedback comments retrieved from the ePortfolios of 149 healthcare students (specialist medicine, general practice, occupational therapy, speech therapy and midwifery) were analysed retrospectively using deductive content analysis. Two structured categorisation matrices, one based on four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and another one on the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional), guided the analysis. Results: The minority of the feedback comments (n = 352; 14.9%) could be considered of high quality because they met all four quality criteria. Most feedback comments were of moderate quality and met only two to three quality criteria. Regarding the CanMEDS roles, the Medical Expert role was most frequently represented in the feedback comments, as opposed to the roles Leader and Health Advocate. Discussion: The results highlighted that providing high-quality feedback is challenging. To respond to these challenges, it is recommended to set up individual and continuous feedback training.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

Van Ostaeyen,

Embo,

Rotsaert

et al. 2023

Perspect Med Educ

View full text Add to dashboard Cite

show abstract

“…7 Although the NEQI includes key quality indicators of narrative assessments, 39 there are other ways to measure narrative quality. 39,57,58 The NEQI reflects narrative quality but does not assess narrative accuracy or account for bias, both of which are critical to providing valid clinical assessment and entrustment decisions. Indeed, gender-based linguistic differences or other implicit biases may exist in otherwise highquality narratives.…”

Section: Discussionmentioning

confidence: 99%

“…Related, our measure of time is constrained to single clerkship rotations and may insufficiently represent longitudinal, trusting relationships thought to facilitate the exchange of useful feedback 7 . Although the NEQI includes key quality indicators of narrative assessments, 39 there are other ways to measure narrative quality 39,57,58 . The NEQI reflects narrative quality but does not assess narrative accuracy or account for bias, both of which are critical to providing valid clinical assessment and entrustment decisions.…”

Section: Discussionmentioning

confidence: 99%

Predictors of faculty narrative evaluation quality in medical school clerkships

et al. 2022

View full text Add to dashboard Cite

Introduction: Narrative approaches to assessment provide meaningful and valid representations of trainee performance. Yet, narratives are frequently perceived as vague, nonspecific and low quality. To date, there is little research examining factors associated with narrative evaluation quality, particularly in undergraduate medical education. The purpose of this study was to examine associations of faculty-and student-level characteristics with the quality of faculty member's narrative evaluations of clerkship students. Methods:The authors reviewed faculty narrative evaluations of 50 students' clinical performance in their inpatient medicine and neurology clerkships, resulting in 165 and 87 unique evaluations in the respective clerkships. The authors evaluated narrative quality using the Narrative Evaluation Quality Instrument (NEQI). The authors used linear mixed effects modelling to predict total NEQI score. Explanatory covariates included the following: time to evaluation completion, number of weeks spent with student, faculty total weeks on service per year, total faculty years in clinical education, student gender, faculty gender, and an interaction term between student and faculty gender.Results: Significantly higher narrative evaluation quality was associated with a shorter time to evaluation completion, with NEQI scores decreasing by approximately 0.3 points every 10 days following students' rotations (p = .004).Additionally, women faculty had statistically higher quality narrative evaluations with NEQI scores 1.92 points greater than men faculty (p = .012). All other covariates were not significant. Conclusions:The quality of faculty members' narrative evaluations of medical students was associated with time to evaluation completion and faculty gender but not faculty experience in clinical education, faculty weeks on service, or the amount of time spent with students. Findings advance understanding on ways to improve the quality of narrative evaluations which are imperative given assessment models that will increase the volume and reliance on narratives.

show abstract

“…Our study uses a quantitative method of evaluating qualitative written narrative feedback; at present there are four validated tools to evaluate the quality of narrative comments in the context of CBME. [10][11][12][13][14][15] When thoughtfully composed, narrative feedback is a personalized commentary on resident performance. 16 Effective feedback has been described as timely, specific, and actionable, with an emphasis on coaching behaviors versus high-stakes assessment.…”

Section: Introductionmentioning

confidence: 99%

Use the right words: evaluating the effect of word choice and word count on quality of narrative feedback in ophthalmology competency-based medical education assessments

Curtis,

Moon,

Hanmore

et al. 2024

Can. Med. Ed. J

View full text Add to dashboard Cite

Background: The purpose of this study was to investigate the effect of word choice on the quality of narrative feedback in ophthalmology resident trainee assessments following the introduction of competency-based medical education at Queen’s University. Methods: Assessment data from July 2017-December 2020 were retrieved from ElentraTM (Integrated Teaching and Learning Platform) and anonymized. Written feedback was assigned a Quality of Assessment for Learning (QuAL) score out of five based on this previously validated rubric. The correlation between QuAL score and specific coaching words was determined using a Spearman’s Rho analysis. Independent samples t-tests were used to compare the QuAL score when a specific word was used, and when it was absent. Results: A total of 1997 individual assessments were used in this analysis. The number of times the identified coaching words were used within a comment was significantly and positively associated with the total QuAL score, with the exception of “next time” (rho=0.039, p=0.082), “read” (rho = 0.036, p = 0.112), “read more” (rho = -0.025, p = 0.256) and “review” (rho = -0.017, p = 0.440). The strongest correlations were for “continue” (rho = 0.182, p < 0.001), “try(ing)” (rho = 0.113, p < 0.001) and “next step” (rho = 0.103, p < 0.001). The mean value of the QuAL score increased when coaching words were used vs. not used with the largest mean difference of 1.44 (p < 0.001) for “reflect”. A clear positive relationship was demonstrated between word count and QuAL score (rho = .556, p < 0.001). Conclusions: The use of certain coaching words in written comments may improve the quality of feedback.

show abstract

Comparing the Quality of Narrative Comments by Rotation Setting

Cited by 4 publications

References 31 publications

A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles

Predictors of faculty narrative evaluation quality in medical school clerkships

Use the right words: evaluating the effect of word choice and word count on quality of narrative feedback in ophthalmology competency-based medical education assessments

Contact Info

Product

Resources

About