Narrative descriptions should replace grades and numerical ratings for clinical performance in medical education in the United States

Abstract

Background: In medical education, evaluation of clinical performance is based almost universally on rating scales for defined aspects of performance and scores on examinations and checklists. Unfortunately, scores and grades do not capture progress and competence among learners in the complex tasks and roles required to practice medicine. While the literature suggests serious problems with the validity and reliability of ratings of clinical performance based on numerical scores, the critical issue is not that judgments about what is observed vary from rater to rater but that these judgments are lost when translated into numbers on a scale. As the Next Accreditation System of the Accreditation Council on Graduate Medical Education (ACGME) takes effect, medical educators have an opportunity to create new processes of evaluation to document and facilitate progress of medical learners in the required areas of competence.Proposal and initial experience: Narrative descriptions of learner performance in the clinical environment, gathered using a framework for observation that builds a shared understanding of competence among the faculty, promise to provide meaningful qualitative data closely linked to the work of physicians. With descriptions grouped in categories and matched to milestones, core faculty can place each learner along the milestones' continua of progress. This provides the foundation for meaningful feedback to facilitate the progress of each learner as well as documentation of progress toward competence.Implications: This narrative evaluation system addresses educational needs as well as the goals of the Next Accreditation System for explicitly documented progress. Educators at other levels of education and in other professions experience similar needs for authentic assessment and, with meaningful frameworks that describe roles and tasks, may also find useful a system built on descriptions of learner performance in actual work settings.Conclusions: We must place medical learning and assessment in the contexts and domains in which learners do clinical work. The approach proposed here for gathering qualitative performance data in different contexts and domains is one step along the road to moving learners toward competence and mastery.

Relatively speaking: contrast effects influence assessors’ scores and narrative feedback

“…have suggested that score variations arise in part because scales may not align with assessors’ thinking, suggesting that ‘disappointing psychometric performance of WBA to date may stem not from disagreements about the performance observed, but from different interpretations of the questions and the scales’. Other authors have suggested that assessments should focus on narrative comments rather than scores . Research into inter‐assessor variability amongst free‐text feedback is far less extensive than has been conducted on assessment scores, although research within occupational psychology indicates that narrative feedback is complex and subject to a variety of influences .…”

Grading Practices and Distributions Across Internal Medicine Clerkships

“…21 It is remarkable, however, that 60% of our respondents did not believe that pass/fail grading was appropriate for clinical clerkships. One reason for this may be that elimination of grades would then force the system to rely on other variables to make high-stakes decisions, likely leading to an even greater reliance on the results of normative United States Medical Licensing Examination Step Exam scores and Subject Exams.…”

Gestalt assessment of online educational resources may not be sufficiently reliable and consistent

“…For example, different inferences about the target demographic among raters may have contributed to variability [27]. Also, with the lack of clear criteria and limited understanding of each number on the 7‑point scale, raters may have been unable to have a clear and consistent strategy when converting their judgments about a resource into a numerical rating [28]. …”

“…There were no clear criteria or anchors defined to convert gestalt opinion into a numerical rating on a scale, which may have contributed to the inconsistency and variability in ratings [28]. While we intentionally kept our anchors vague so that we might best approximate a reader’s innate gestalt, it is possible that a more directed question (e. g. ‘Would you recommend this to a postgraduate year 1 resident?’) would yield a more reliable answer.…”