Narrative Assessments in Higher Education: A Scoping Review to Identify Evidence-Based Quality Indicators

Abstract

PurposeNarrative comments are increasingly used in assessment to document trainees' performance and to make important decisions about academic progress. However, little is known about how to document the quality of narrative comments, since traditional psychometric analysis cannot be applied. The authors aimed to generate a list of quality indicators for narrative comments, to identify recommendations for writing high-quality narrative comments, and to document factors that influence the quality of narrative comments used in assessments in higher education. MethodThe authors conducted a scoping review according to Arksey & O'Malley's framework. The search strategy yielded 690 articles from 6 databases.

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education

“…Assessment narratives contain contextualized details about trainee performance that inform learning and promotion decisions. [1][2][3][4][5] High-quality learner assessments are essential for programmatic assessment, a key component of competency-based medical education (CBME) that requires the integration of assessments completed by multiple supervisors in varying contexts over time to effectively characterize the complexity of clinical competence. 6,7 When narrative comments are of low quality, confidence in high-stakes decisions regarding trainee performance and progression declines.…”

“…8,9 Educators have developed tools to evaluate the quality of narrative assessment comments in medical education. 5 The System for Improving and Measuring Procedural Learning (SIMPL), 10 Narrative Evaluation Quality Instrument (NEQI), 11 Evaluation of Feedback Captured Tool, 12 and Quality of Assessment for Learning (QuAL) 13 tools are used to rate the quality of narrative comments with the goal of supporting and quantifying improvement. For example, the QuAL score is a short 3-item tool with evidence for rating the utility of short narrative comments, specifically in the context of CBME.…”

Predictors of faculty narrative evaluation quality in medical school clerkships

“…Related, our measure of time is constrained to single clerkship rotations and may insufficiently represent longitudinal, trusting relationships thought to facilitate the exchange of useful feedback. 7 Although the NEQI includes key quality indicators of narrative assessments, 39 there are other ways to measure narrative quality. 39,57,58 The NEQI reflects narrative quality but does not assess narrative accuracy or account for bias, both of which are critical to providing valid clinical assessment and entrustment decisions.…”

“…7 Although the NEQI includes key quality indicators of narrative assessments, 39 there are other ways to measure narrative quality. 39,57,58 The NEQI reflects narrative quality but does not assess narrative accuracy or account for bias, both of which are critical to providing valid clinical assessment and entrustment decisions. Indeed, gender-based linguistic differences or other implicit biases may exist in otherwise highquality narratives.…”

“…Prior studies have established validity evidence of the NEQI, including its ability to reliably differentiate narrative evaluation quality in undergraduate medical education settings. 8,29 The NEQI assesses narrative quality along three dimensions, several of which have been recognised as key narrative quality indicators in a recent scoping review 39 : number of performance domains commented on, specificity of comments, and overall usefulness. Each NEQI dimension generates a subcomponent score of 0 to 4, with a total score ranging from 0 to 12.…”