Artificial Intelligence and Patient Education: Examining the Accuracy and Reproducibility of Responses to Nutrition Questions Related to Inflammatory Bowel Disease by GPT-4

Abstract

Background and AimsGenerative Pre-trained Transformer-4 (GPT-4) is a large language model (LLM) trained on a variety of topics, including the medical literature. Nutrition plays a critical role in managing inflammatory bowel disease (IBD), with an unmet need for nutrition-related patient education resources. The aim of this study is to examine the accuracy and reproducibility of responses by GPT-4 to patient nutrition questions related to IBD.MethodsQuestions were curated from adult IBD clinic visits, Facebook, and Reddit. Two IBD-focused registered dieticians independently graded the accuracy and reproducibility of GPT-4’s responses while a third senior IBD-focused registered dietitian arbitrated. To ascertain reproducibility, each question was inputted twice into the model. Descriptive analysis is presented as counts and proportions.ResultsIn total, 88 questions were included. The model provided correct responses to 73/88 questions (83.0%), with 61 (69.0%) graded as comprehensive. A total of 15/88 (17%) responses were graded as mixed with correct and incorrect/outdated data. When examined by category, the model provided comprehensive responses to 10 (62.5%) questions related to “Nutrition and diet needs for surgery”, 12 (92.3%) “Tube feeding and parenteral nutrition”, 11 (64.7%) “General diet questions”, 10 (50%) “Diet for reducing symptoms/inflammation” and 18 (81.8%) “Micronutrients/supplementation needs”. The model provided reproducible responses to 81/88 (92.0%) questions.ConclusionGPT-4 provided comprehensive responses to the majority of questions, demonstrating the promising potential of LLMs as supplementary tools for IBD patients seeking nutrition-related information. However, 17% of responses contained incorrect information, highlighting the need for continuous refinement and validation of LLMs prior to incorporation into clinical practice. Future studies should focus on leveraging LLMs to enhance patient outcomes. Furthermore, efforts promoting patient and healthcare professional proficiency in using LLMs are essential to maximize their efficacy and facilitate personalized care.

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

“…Table 1 summarizes the characteristics of the analyzed studies, including their setting, results, and conclusions. One study (n=1/89, 1.1%) was published in 2022 24 , 84 (n=84/89, 94.4%) in 2023 13,25–107 , and 4 (n=4/89, 4.5%) in 2024 108–111 (all of which were peer-reviewed publications of preprints published in 2023). Most studies were quantitative non-randomized (n=84/89, 94.4%) 13,25–27,29–101,103,104,106,107,109–111 , 4 (n=4/89, 4.5%) 28,102,105,108 had a qualitative study design, and one (n=1/89, 1.1%) 24 was quantitative randomized according to the MMAT 2018 criteria.…”

“…The authors were primarily affiliated with institutions in the United States (n=47 of 122 different countries identified per publication, 38.5%), followed by Germany (n=11/122, 9%), Turkey (n=7/122, 5.7%), the United Kingdom (n=6/122, 4.9%), China/Australia/Italy (n=5/122, 4.1%, respectively), and 24 (n=36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3...…”

“…Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3.2%) 13,83,84 , Large Language Model Meta-AI (LLaMA; n=3/124, 2.4%) 55 , or Claude by Anthropic (n=1/124, 0.8%) 55 . The majority of applications were p...…”

“…Most reports evaluated LLMs in English (n=88/89, 98.9%) 13,24–103,105–111 , followed by Arabic (n=2/84, 2.3%) 32,104 , Mandarin (n=2/84, 2.3%) 36,75 , and Korean or Spanish (n=1/89, 1.1%, respectively) 75 . The top-five specialties studied were ophthalmology (n=10/89, 11.2%) 37,40,48,51,65,74,97,98,100,101 , gastro-enterology (n=9/89, 10.1%) 25,32,34,36,39,61,62,72,96 , head and neck surgery/otolaryngology (n=8/89, 9%) 35,42,56,64,66,76,78,79 , and radiology 59,70,88–90,110 or plastic surgery 45,47,49,102,107,108 (n=6/89, 6.7%, respectively). A schematic illustration of the identified concepts of LLM applications in patient care is shown in Figure 2.…”

“…One study (n=1/89, 1.1%) was published in 2022 24 , 84 (n=84/89, 94.4%) in 2023 13, , and 4 (n=4/89, 4.5%) in 2024 [108][109][110][111] (all of which were peer-reviewed publications of preprints published in 2023). Most studies were quantitative non-randomized (n=84/89, 94.4%) 13,[25][26][27]103,104,106,107,[109][110][111] , 4 (n=4/89, 4.5%) 28,102,105,108 had a qualitative study design, and one (n=1/89, 1.1%) 24 was quantitative randomized according to the MMAT 2018 criteria. However, the LLM outputs were often first analyzed quantitatively but followed by a qualitative analysis of certain responses.…”