Validation of ChatGPT 3.5 as a Tool to Optimize Readability of Patient-facing Craniofacial Education Materials

Vallurupalli, Medha; Shah, Nikhil D.; Vyas, Raj M.

doi:10.1097/gox.0000000000005575

Cited by 4 publications

(3 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, one study identified that in plastic surgery, 50.2% of patients had an education level lower than high school and that 48% of attendings, residents, and PAs said it was challenging to make patients follow postoperative instructions [ 34 ]. Moreover, patient materials in plastic surgery are often above the recommended average reading level of 6th to 8th grade and may be too difficult for the average patient [ 34 , 35 , 36 , 37 ].…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery

Gomez-Cabello,

Borna,

Pressman

et al. 2024

Healthcare

View full text Add to dashboard Cite

Since their release, the medical community has been actively exploring large language models’ (LLMs) capabilities, which show promise in providing accurate medical knowledge. One potential application is as a patient resource. This study analyzes and compares the ability of the currently available LLMs, ChatGPT-3.5, GPT-4, and Gemini, to provide postoperative care recommendations to plastic surgery patients. We presented each model with 32 questions addressing common patient concerns after surgical cosmetic procedures and evaluated the medical accuracy, readability, understandability, and actionability of the models’ responses. The three LLMs provided equally accurate information, with GPT-3.5 averaging the highest on the Likert scale (LS) (4.18 ± 0.93) (p = 0.849), while Gemini provided significantly more readable (p = 0.001) and understandable responses (p = 0.014; p = 0.001). There was no difference in the actionability of the models’ responses (p = 0.830). Although LLMs have shown their potential as adjunctive tools in postoperative patient care, further refinement and research are imperative to enable their evolution into comprehensive standalone resources.

show abstract

Section: Discussionmentioning

confidence: 99%

“…Vallurupalli et al [ 37 ] successfully used ChatGPT-3.5 to significantly simplify traditional patient education materials for craniofacial surgery by 3 FKGL points. Interestingly, in our study, ChatGPT-3.5 and 4’s average FKGL scores were far superior to the recommended reading level.…”

Section: Discussionmentioning

confidence: 99%

Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery

Gomez-Cabello,

Borna,

Pressman

et al. 2024

Healthcare

View full text Add to dashboard Cite

show abstract

“…Plastic surgery is a field of innovation where new techniques and research findings regularly emerge to improve patient-centered outcomes [ 14 , 15 , 16 ]. LLMs have been used in aesthetic surgery, general plastic surgery, craniofacial, microsurgery, and hand surgery, where they have displayed adequate knowledge for research proposals, patient counseling, clinical decision-making, and intraoperative support [ 2 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ]. To evaluate ChatGPT’s knowledge of plastic surgery, Humar et al [ 28 ] prompted the questions from the 2022 In-Service Examination, where the model surprisingly ranked in the 49th percentile for 1st-year integrated plastic surgery residents.…”

Section: Introductionmentioning

confidence: 99%

Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini

Gomez-Cabello,

Borna,

Pressman

et al. 2024

Medicina

View full text Add to dashboard Cite

Background and Objectives: Large language models (LLMs) are emerging as valuable tools in plastic surgery, potentially reducing surgeons’ cognitive loads and improving patients’ outcomes. This study aimed to assess and compare the current state of the two most common and readily available LLMs, Open AI’s ChatGPT-4 and Google’s Gemini Pro (1.0 Pro), in providing intraoperative decision support in plastic and reconstructive surgery procedures. Materials and Methods: We presented each LLM with 32 independent intraoperative scenarios spanning 5 procedures. We utilized a 5-point and a 3-point Likert scale for medical accuracy and relevance, respectively. We determined the readability of the responses using the Flesch–Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) score. Additionally, we measured the models’ response time. We compared the performance using the Mann–Whitney U test and Student’s t-test. Results: ChatGPT-4 significantly outperformed Gemini in providing accurate (3.59 ± 0.84 vs. 3.13 ± 0.83, p-value = 0.022) and relevant (2.28 ± 0.77 vs. 1.88 ± 0.83, p-value = 0.032) responses. Alternatively, Gemini provided more concise and readable responses, with an average FKGL (12.80 ± 1.56) significantly lower than ChatGPT-4′s (15.00 ± 1.89) (p < 0.0001). However, there was no difference in the FRE scores (p = 0.174). Moreover, Gemini’s average response time was significantly faster (8.15 ± 1.42 s) than ChatGPT’-4′s (13.70 ± 2.87 s) (p < 0.0001). Conclusions: Although ChatGPT-4 provided more accurate and relevant responses, both models demonstrated potential as intraoperative tools. Nevertheless, their performance inconsistency across the different procedures underscores the need for further training and optimization to ensure their reliability as intraoperative decision-support tools.

show abstract

Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model

Swisher,

Wu,

Liu

et al. 2024

Otolaryngol.--head neck surg.

View full text Add to dashboard Cite

ObjectiveTo use an artificial intelligence (AI)‐powered large language model (LLM) to improve readability of patient handouts.Study DesignReview of online material modified by AI.SettingAcademic center.MethodsFive handout materials obtained from the American Rhinologic Society (ARS) and the American Academy of Facial Plastic and Reconstructive Surgery websites were assessed using validated readability metrics. The handouts were inputted into OpenAI's ChatGPT‐4 after prompting: “Rewrite the following at a 6th‐grade reading level.” The understandability and actionability of both native and LLM‐revised versions were evaluated using the Patient Education Materials Assessment Tool (PEMAT). Results were compared using Wilcoxon rank‐sum tests.ResultsThe mean readability scores of the standard (ARS, American Academy of Facial Plastic and Reconstructive Surgery) materials corresponded to “difficult,” with reading categories ranging between high school and university grade levels. Conversely, the LLM‐revised handouts had an average seventh‐grade reading level. LLM‐revised handouts had better readability in nearly all metrics tested: Flesch‐Kincaid Reading Ease (70.8 vs 43.9; P < .05), Gunning Fog Score (10.2 vs 14.42; P < .05), Simple Measure of Gobbledygook (9.9 vs 13.1; P < .05), Coleman‐Liau (8.8 vs 12.6; P < .05), and Automated Readability Index (8.2 vs 10.7; P = .06). PEMAT scores were significantly higher in the LLM‐revised handouts for understandability (91 vs 74%; P < .05) with similar actionability (42 vs 34%; P = .15) when compared to the standard materials.ConclusionPatient‐facing handouts can be augmented by ChatGPT with simple prompting to tailor information with improved readability. This study demonstrates the utility of LLMs to aid in rewriting patient handouts and may serve as a tool to help optimize education materials.Level of EvidenceLevel VI.

show abstract

Validation of ChatGPT 3.5 as a Tool to Optimize Readability of Patient-facing Craniofacial Education Materials

Cited by 4 publications

References 22 publications

Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery

Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery

Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini

Enhancing Health Literacy: Evaluating the Readability of Patient Handouts Revised by ChatGPT's Large Language Model

Contact Info

Product

Resources

About