Assessing the Ability of GPT to Generate Illness Scripts: An Evaluation Study

Abstract

BackgroundIllness scripts, which are structured summaries of clinical knowledge concerning diseases, are crucial in disease prediction and problem representation during clinical reasoning. Clinicians iteratively enhance their illness scripts through their clinical practice. Because illness scripts are unique to each physician, no systematic summary of specific examples of illness scripts has been reported.ObjectiveGenerative artificial intelligence (AI) stands out as an educational aid in continuing medical education. The effortless creation of a typical illness script by generative AI could enhance the comprehension of disease concepts and increase diagnostic accuracy. This study investigated whether generative AI possesses the capability to generate illness scripts.MethodsWe used ChatGPT, a generative AI, to create illness scripts for 184 diseases based on the diseases and conditions integral to the National Model Core Curriculum for undergraduate medical education (2022 revised edition) and primary care specialist training in Japan. Three physicians applied a three-tier grading scale: “A” if the content of each disease’s illness script proves sufficient for training medical students, “B” if it is partially lacking but acceptable, and “C” if it is deficient in multiple respects. Moreover, any identified deficiencies in the illness scripts were discussed during the evaluation process.ResultsLeveraging ChatGPT, we successfully generated each component of the illness script for 184 diseases without any omission. The illness scripts received “A,” “B,” and “C” ratings of 56.0% (103/184), 28.3% (52/184), and 15.8% (29/184), respectively.ConclusionUseful illness scripts were seamlessly and instantaneously created by ChatGPT using prompts appropriate for medical students. The technology-driven illness script is a valuable tool for introducing medical students to disease conceptualization.