Examining the Threat of ChatGPT to the Validity of Short Answer Assessments in an Undergraduate Medical Program

Abstract

OBJECTIVES ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance ( P = .008, η2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.

Examining the Efficacy of ChatGPT in Marking Short-Answer Assessments in an Undergraduate Medical Program

“…The frequency of changes between scoring categories (34-57%) suggests that relying solely on AI-based grading could sometimes overlook nuances that would be critical in a medical educational context. However, while not formally tested in this study, there also exists some level of inter-rater variability with independent human tutors; our group formally investigated in a previous work and found a Cronbach alpha value of 0.816 for a team of six human assessors on past student-generated CAE responses [29].…”

A scoping review on how generative artificial intelligence transforms assessment in higher education

“…This poses comprehension challenges for students with low abilities (Guo & Wang, 2023). In addition, some teachers noticed that ChatGPT might use different evaluation criteria from their own, and its lack of specific knowledge about the class and students could lead to inappropriate feedback (Morjaria et al, 2023). These limitations indicated that although ChatGPT seemed to be powerful, it could not replace teacher feedback.…”

“…For example, GenAI can be used to advance writing such as proofreading, critique, and editing (Currie et al, 2023), create personalized assessments, and simulate conversations (Cheung et al, 2023;Currie et al, 2023). Therefore, the teachers were encouraged to use alternative grading practices, like incorporating nontraditional, authentic assessments that are difficult for AI to replicate without prompting (Chaudhry et al, 2023;Fuchs et al, 2023;Morjaria et al, 2023;Overono & Ditta, 2023;Perkins, 2023). In other words, the students' language can be self-assessed through GenAI, while the students' ideas and logical thinking can be assessed by teachers.…”

“…The use of AI tools like ChatGPT in education requires rethinking assessment policies (Gorichanaz, 2023;Morjaria et al, 2023), as traditional examinations and assignments may become obsolete due to AI-generated content (Geerling et al, 2023;Gorichanaz, 2023). When making policies for assessment, educational institutions should carefully rethink what assessment approaches are needed to ensure academic integrity and meaningful learning (Perkins, 2023;Rajabi et al, 2023).…”

Assessing the Ability of a Large Language Model to Score Free-Text Medical Student Clinical Notes: Quantitative Study

“…ChatGPT has been compared to human raters in terms of grading short-answer pre-clerkship medical questions. The ChatGPT-human Spearman correlations for a single assessor ranged from 0.6 to 0.7 [12].…”