Using Automated Scoring to Evaluate Written Responses in English and French on a High-Stakes Clinical Competency Examination

Latifi, Syed; Gierl, Mark J.; Boulais, André-Philippe; Champlain, André F. De

doi:10.1177/0163278715605358

Cited by 16 publications

(10 citation statements)

References 15 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…AI is also used in assessment of learners like in assignment grading, automated essay scoring using clinical decision-making questions, evaluation of basic laparoscopic skills, grading of student case summaries, attendance tracking to name a few. [13][14][15] Main barrier for AI limited use in assessments is lack of digitalization which impact on meeting the data pool requirements to develop AI based system. This limitation in much more pronounced in our educational institutions where we are using mostly non-digital tools for different domains including teaching, assessment, curriculum and evaluation.…”

mentioning

confidence: 99%

Artificial intelligence in medical education: Are we ready for it?

Imran

Jawaid²

2020

Pak J Med Sci

View full text Add to dashboard Cite

doi: https://doi.org/10.12669/pjms.36.5.3042 How to cite this:Imran N, Jawaid M. Artificial intelligence in medical education: Are we ready for it? Pak J Med Sci. 2020;36(5):857-859. doi: https://doi.org/10.12669/pjms.36.5.3042 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

show abstract

mentioning

confidence: 99%

Artificial intelligence in medical education: Are we ready for it?

Imran

Jawaid²

2020

Pak J Med Sci

View full text Add to dashboard Cite

show abstract

“…Despite the promise of ASAG using NLP and ML, the use of ASAG for PN grading is difficult. 6,11 Relevant clinical concepts are clinical case-specific, and effective training for such a task requires a large number of PNs with annotated phrase instances (phrases paired with graded checklist items) to train supervised ML algorithms. In addition, learners often use inconsistent and nonstandard abbreviations (e.g., "h" or "h/o" or "hx" all meaning "history of "), and abbreviations may vary in meaning based on case context (e.g., CP as "chest pain" or "cerebral palsy").…”

Section: Discussionmentioning

confidence: 99%

Automated Patient Note Grading: Examining Scoring Reliability and Feasibility

Bond,

Zhou,

Bhat

et al. 2023

Academic Medicine

View full text Add to dashboard Cite

Purpose: Scoring post-encounter patient notes (PNs) yields significant insights into student performance, but the resource intensity of scoring limits its use. Recent advances in natural language processing (NLP) and machine learning (ML) allow the application of automated short answer grading (ASAG) for this task. This retroactive study evaluates the psychometric characteristics and reliability of an ASAG system for PNs, and secondarily considered factors contributing to implementation, including feasibility and case-specific phrase annotation required to tune the system for a new case. Method: PNs from standardized patient (SP) cases within a graduation competency exam were used to train the ASAG system, applying a feed-forward neural networks algorithm for scoring. Using faculty phrase-level annotation, 10 PNs per case were required to tune the ASAG system. After tuning, ASAG item-level ratings for 20 notes were compared across ASAG-faculty (4 cases, 80 pairings) and ASAG-non-faculty (2 cases, 40 pairings). Psychometric characteristics were examined using item analysis and Cronbach’s alpha. Inter-rater reliability (IRR) was examined using kappa. Results: ASAG-scores demonstrated sufficient variability in differentiating learner PN performance and high IRR between machine-human ratings. Across all items the ASAG-faculty scoring mean kappa was 0.83 (SE±0.02). The ASAG-non-faculty pairings kappa was 0.83 (SE±0.02). The ASAG scoring demonstrated high item discrimination. Internal consistency reliability values at the case level ranged from a Cronbach’s alpha of 0.65 to 0.77. Faculty time cost to train and supervise non-faculty raters for 4 cases is approximately $1,856. Faculty cost to tune the ASAG system is approximately $928. Conclusions: NLP-based automated scoring of PNs demonstrated a high degree of reliability and psychometric confidence for use as learner feedback. The small number of phrase level annotations required to tune the system to a new case enhances feasibility. ASAG-enabled PN scoring has broad implications for improving feedback in case-based learning contexts in medical education.

show abstract

“…Automated feedback use has been reported in medical education studies ranging from simple online multiple choice tests 6 to clinical competency essay marking. 7 It has also been consistently used to good effect in the field of computer coding education 8 where users submit their code and receive feedback designed to identify aspects that need improving. Planning metrics software works in a similar manner by providing a rapid overview of student performance across a range of parameters to highlight the most challenging aspects and focus efforts accordingly.…”

Section: Journal Of Radiotherapy In Practicementioning

confidence: 99%

“…Although these applications are designed for clinical use as plan evaluation tools there is potential academic value in providing automated feedback to students regarding plan quality. Automated feedback use has been reported in medical education studies ranging from simple online multiple choice tests 6 to clinical competency essay marking 7 . It has also been consistently used to good effect in the field of computer coding education 8 where users submit their code and receive feedback designed to identify aspects that need improving.…”

Section: Introductionmentioning

confidence: 99%

Use of planning metrics software for automated feedback to radiotherapy students

2016

View full text Add to dashboard Cite

Background and purposePre-registration teaching of radiotherapy planning in a non-clinical setting should allow students the opportunity to develop clinical decision-making skills. Students frequently struggle with their ability to prioritise and optimise multiple objectives when producing a clinically acceptable plan. Emerging software applications providing quantitative assessment of plan quality are designed for clinical use but may have value for teaching these skills. This project aimed to evaluate the potential value of automated feedback to second year BSc (Hons) Radiotherapy students.Materials and methodsAll 26 students studying a pre-registration radiotherapy planning module were provided with automated prediction of relative feasibility for left lung tumour planning targets by planning metrics software. Students were also provided with interim quantitative reports during the development of their plan. Student perceptions of the software were gathered using an anonymous questionnaire. Independent blinded marking of plans was performed after module completion and analysed for correlation with software-assigned marks.ResultsIn total, 25 plans were utilised for marking comparison and 16 students submitted feedback relating to the software. Overall, student feedback was positive regarding the software. A ‘strong’ Spearman’s rank-order correlation (rs=0·7165) was evident between human and computer marks (p=0·000055).ConclusionsAutomated software is capable of providing useful feedback to students as a teaching aid, in particular with regard to relative feasibility of goals. The strong correlation between human and computer marks suggests a role in benchmarking or moderation; however, the narrow scope of assessment parameters suggests value as an adjunct and not a replacement to human marking.

show abstract

Using Automated Scoring to Evaluate Written Responses in English and French on a High-Stakes Clinical Competency Examination

Cited by 16 publications

References 15 publications

Artificial intelligence in medical education: Are we ready for it?

Artificial intelligence in medical education: Are we ready for it?

Automated Patient Note Grading: Examining Scoring Reliability and Feasibility

Use of planning metrics software for automated feedback to radiotherapy students

Contact Info

Product

Resources

About