Natural Language Processing and Assessment of Resident Feedback Quality

Solano, Quintin P.; Hayward, Laura; Chopra, Zoey; Quanstrom, Kathryn; Kendrick, Daniel E.; Abbott, Kenneth L.; Kunzmann, Marcus; Ahle, Samantha L.; Schuller, Mary C.; Ötleş, Erkin; George, Brian C.

doi:10.1016/j.jsurg.2021.05.012

Cited by 26 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This study builds on prior endeavors to use NLP as a tool to assist residency programs in the ongoing evaluation of trainee progress. While several prior studies address NLP to predict the quality or utility of evaluations, 10,19,20 our results suggest that it may be as prudent to target what faculty write about in their evaluations, rather than only how they write them. Comments that were not useful or about demographic content made up more than 30% of the data.…”

Section: Ics (100%)mentioning

confidence: 63%

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

Booth¹,

Ross²,

Cronin³

et al. 2022

Academic Medicine

View full text Add to dashboard Cite

PurposeFaculty feedback on trainees is critical to guiding trainee progress in a competencybased medical education framework. The authors aimed to develop and evaluate a Natural Language Processing (NLP) algorithm that automatically categorizes narrative feedback into corresponding Accreditation Council for Graduate Medical Education Milestone 2.0 subcompetencies. MethodTen academic anesthesiologists analyzed 5,935 narrative evaluations on anesthesiology trainees at 4 graduate medical education (GME) programs between July 1, 2019, and June 30, 2021. Each sentence (n = 25,714) was labeled with the Milestone 2.0 subcompetency that best captured its content or was labeled as demographic or not useful. Inter-rater agreement was assessed by Fleiss' Kappa. The authors trained an NLP model to predict feedback subcompetencies using data from 3 sites and evaluated its performance at a fourth site. Performance metrics included area under the receiver operating characteristic curve (AUC), positive predictive value, sensitivity, F1, and calibration curves. The model was implemented at 1 site in a self-assessment exercise. ResultsFleiss' Kappa for subcompetency agreement was moderate (0.44). Model performance was good for professionalism, interpersonal and communication skills, and practicebased learning and improvement (AUC 0.79, 0.79, and 0.75, respectively). Subcompetencies within medical knowledge and patient care ranged from fair to excellent (AUC 0.66-0.84 and 0.63-0.88, respectively). Performance for systems-based practice was poor (AUC 0.59). Performances for demographic and not useful categories were excellent (AUC 0.87 for both). In approximately 1 minute, the model interpreted several hundred evaluations and produced individual trainee reports with organized feedback to guide a self-assessment exercise. The model was built into a webbased application. ConclusionsThe authors developed an NLP model that recognized the feedback language of anesthesiologists across multiple GME programs. The model was operationalized in a self-assessment exercise. It is a powerful tool which rapidly organizes large amounts of narrative feedback.

show abstract

Section: Ics (100%)mentioning

confidence: 63%

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

Booth¹,

Ross²,

Cronin³

et al. 2022

Academic Medicine

View full text Add to dashboard Cite

show abstract

“…[22][23][24][25] Critically, the few NLP studies focused on evaluating the quality of narrative comments have not incorporated a standardized tool (such as the QuAL score) to define quality. 26,27…”

Section: Discussionmentioning

confidence: 99%

“…Ötleş et al 27 assessed 600 narrative comments from 3 different surgical training programs and created an NLP model for discriminating high- from low-quality comments with an accuracy of 0.64 27 . Solano et al 26 similarly developed an NLP model for 2,416 surgical resident feedback comments with an accuracy, sensitivity, and specificity of 83%, 37%, and 97%, respectively. These metrics are for a 2-class rating system only; both studies used the method of quality outlined by Ahle et al 41 in which 2 raters code comments into “effective, mediocre, ineffective, or other” then subsequently combined these codes into “high-quality” and “low-quality” groups 41 …”

Section: Discussionmentioning

confidence: 99%

“…Within medical education, NLP applications have focused primarily on predicting clinical performance 22–25 . Critically, the few NLP studies focused on evaluating the quality of narrative comments have not incorporated a standardized tool (such as the QuAL score) to define quality 26,27 . Therefore, we developed an NLP model for applying the QuAL score to narrative supervisor comments.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education

Spadafore,

Yilmaz,

Rally

et al. 2024

Acad Med

View full text Add to dashboard Cite

Purpose Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. Method Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019–2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. Results All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. Conclusions This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

show abstract

“…Another potential application of LLMs in surgical training is the extraction and summarization of key insights from large volumes of feedback data 31 . By analyzing feedback from multiple trainers or across numerous training instances, LLMs could identify recurring themes or patterns and provide students with concise, actionable summaries.…”

Section: Introductionmentioning

confidence: 99%

Innovations in surgical training: exploring the role of artificial intelligence and large language models (LLM)

VARAS,

CORONEL,

VILLAGRÁN

et al. 2023

Rev. Col. Bras. Cir.

View full text Add to dashboard Cite

The landscape of surgical training is rapidly evolving with the advent of artificial intelligence (AI) and its integration into education and simulation. This manuscript aims to explore the potential applications and benefits of AI-assisted surgical training, particularly the use of large language models (LLMs), in enhancing communication, personalizing feedback, and promoting skill development. We discuss the advancements in simulation-based training, AI-driven assessment tools, video-based assessment systems, virtual reality (VR) and augmented reality (AR) platforms, and the potential role of LLMs in the transcription, translation, and summarization of feedback. Despite the promising opportunities presented by AI integration, several challenges must be addressed, including accuracy and reliability, ethical and privacy concerns, bias in AI models, integration with existing training systems, and training and adoption of AI-assisted tools. By proactively addressing these challenges and harnessing the potential of AI, the future of surgical training may be reshaped to provide a more comprehensive, safe, and effective learning experience for trainees, ultimately leading to better patient outcomes. .

show abstract

Natural Language Processing and Assessment of Resident Feedback Quality

Cited by 26 publications

References 20 publications

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

Competency-Based Assessments: Leveraging Artificial Intelligence to Predict Subcompetency Content

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education

Innovations in surgical training: exploring the role of artificial intelligence and large language models (LLM)

Contact Info

Product

Resources

About