The Quality of Written Feedback by Attendings of Internal Medicine Residents

Abstract

BACKGROUND: Attending evaluations are commonly used to evaluate residents. OBJECTIVES: Evaluate the quality of written feedback of internal medicine residents. DESIGN: Retrospective. PARTICIPANTS: Internal medicine residents and faculty at the Medical College of Wisconsin from 2004 to 2012. MAIN MEASURES: From monthly evaluations of residents by attendings, a randomly selected sample of 500 written comments by attendings were qualitatively coded and rated as high-, moderate-, or low-quality feedback by two independent coders with good inter-rater reliability (kappa: 0.94). Small group exercises with residents and attendings also coded the utterances as high, moderate, or low quality and developed criteria for this categorization. In-service examination scores were correlated with written feedback. KEY RESULTS: There were 228 internal medicine residents who had 6,603 evaluations by 334 attendings. Among 500 randomly selected written comments, there were 2,056 unique utterances: 29 % were coded as nonspecific statements, 20 % were comments about resident personality, 16 % about patient care, 14 % interpersonal communication, 7 % medical knowledge, 6 % professionalism, and 4 % each on practice-based learning and systems-based practice. Based on criteria developed by group exercises, the majority of written comments were rated as moderate quality (65 %); 22 % were rated as high quality and 13 % as low quality. Attendings who provided high-quality feedback rated residents significantly lower in all six of the Accreditation Council for Graduate Medical Education (ACGME) competencies (p <0.0005 for all), and had a greater range of scores. Negative comments on medical knowledge were associated with lower in-service examination scores. CONCLUSIONS: Most attending written evaluation was of moderate or low quality. Attendings who provided highquality feedback appeared to be more discriminating, providing significantly lower ratings of residents in all six ACGME core competencies, and across a greater range. Attendings' negative written comments on medical knowledge correlated with lower in-service training scores.

Supervisor-trainee continuity and the quality of work-based assessments

“…25,26 The mean CCERR scores for all three groups in our study were lower, suggesting that the quality of DECs completed by supervisors fell below a standard that would be considered to represent 'average' quality. Although this is disappointing, our results are congruent with those of other studies which demonstrate that clinical supervisors consistently fail to complete WBAs to an adequate standard, as represented by similarly low mean CCERR scores (range: [15][16][17][18][19][20][21][22]. 11,12,31 Given the central role of WBA in current postgraduate competency-based curricula, [32][33][34] determining how to improve the quality of these assessments has become paramount.…”

“…A total of 383 DECs (CTT: n = 183; Non-CTT: n = 100; Off-service: n = 100) were scored using the CCERR by two raters. Clinical teaching team pairs worked a mean AE SD of 9.15 AE 3.92 (range: [5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20] shifts together during the academic year. Supervisors worked a mean AE SD of 2.43 AE 0.84 shifts (range: 1-4) with Non-CTT residents and 1.09 AE 0.29 shifts (range: 1-2) with Off-service residents.…”

“…3,5,[14][15][16] The resultant effects are assessments that contain mediocre comments and non-discriminatory ratings. 3,5,17 This is of particular concern when supervisors report judgements of underperforming trainees as these barriers to documenting high-quality assessments may contribute to the 'failure to fail' phenomenon. 7,16 One proposed method of addressing the issue of assessment quality has been to increase the interaction between supervisor and trainee by ensuring that they work together more frequently.…”

What are the associations between the quantity of faculty evaluations and residents’ perception of quality feedback?

“…Peer evaluations of residents might provide an even greater value than those from faculty [20], [21]. Holmes and others suggested a structured method to provide formative feedback at the end of a clinical experience [22], [23]. In psychiatry programs, direct observation of clinical work is an excellent opportunity to provide formative feedback to residents.…”

Capsule Commentary on Jackson et al., The Quality of Written Feedback by Attendings of Internal Medicine Residents

“…I n this retrospective study by Jackson et al, 1 500 evaluations of internal medicine residents by their attending physicians were randomly selected (from 6,603 available evaluations), and the written feedback coded and categorized. Criteria for effective feedback developed by small groups were independently applied by group participants and investigators to rate the feedback as low, moderate, or high quality.…”