Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making

Abstract

Background Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student’s performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making. Methods In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner’s interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data. Results The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student’s performance and quality of narrative feedback. Conclusion The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these “difficult” portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education

“…38 This model is a strong step toward our goal of identifying low-quality comments (and raters) for faculty development and programmatic improvement. 8,9,39 Additionally, trainees have identified that collecting EPA assessments from supervisors represents a significant burden 40 ; we owe it to trainees to provide them with valuable and useful feedback if we are adding to their already busy workload.…”

“…6,7 When narrative comments are of low quality, confidence in high-stakes decisions regarding trainee performance and progression declines. 8,9 Educators have developed tools to evaluate the quality of narrative assessment comments in medical education. 5 The System for Improving and Measuring Procedural Learning (SIMPL), 10 Narrative Evaluation Quality Instrument (NEQI), 11 Evaluation of Feedback Captured Tool, 12 and Quality of Assessment for Learning (QuAL) 13 tools are used to rate the quality of narrative comments with the goal of supporting and quantifying improvement.…”

“…The model was most effective in identifying narrative comments that lacked a suggestion for improvement, which is a key element of effective feedback 37 and something that is commonly missing from narrative comments 38 . This model is a strong step toward our goal of identifying low-quality comments (and raters) for faculty development and programmatic improvement 8,9,39 . Additionally, trainees have identified that collecting EPA assessments from supervisors represents a significant burden 40 ; we owe it to trainees to provide them with valuable and useful feedback if we are adding to their already busy workload.…”

“…High-quality learner assessments are essential for programmatic assessment, a key component of competency-based medical education (CBME) that requires the integration of assessments completed by multiple supervisors in varying contexts over time to effectively characterize the complexity of clinical competence 6,7 . When narrative comments are of low quality, confidence in high-stakes decisions regarding trainee performance and progression declines 8,9 …”

Correction: Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making

“…Following publication of the original article [ 1 ], the authors informed us that due to an error in the proofing process, one reference was not correctly in the reference list and the entire reference list needs to be updated.…”