Evaluating the Accuracy of ChatGPT and Google BARD in Fielding Oculoplastic Patient Queries: A Comparative Study on Artificial versus Human Intelligence

Abstract

Purpose: This study evaluates and compares the accuracy of responses from 2 artificial intelligence platforms to patients’ oculoplastics-related questions. Methods: Questions directed toward oculoplastic surgeons were collected, rephrased, and input independently into ChatGPT-3.5 and BARD chatbots, using the prompt: “As an oculoplastic surgeon, how can I respond to my patient’s question?.” Responses were independently evaluated by 4 experienced oculoplastic specialists as comprehensive, correct but inadequate, mixed correct and incorrect/outdated data, and completely incorrect. Additionally, the empathy level, length, and automated readability index of the responses were assessed. Results: A total of 112 patient questions underwent evaluation. The rates of comprehensive, correct but inadequate, mixed, and completely incorrect answers for ChatGPT were 71.4%, 12.9%, 10.5%, and 5.1%, respectively, compared with 53.1%, 18.3%, 18.1%, and 10.5%, respectively, for BARD. ChatGPT showed more empathy (48.9%) than BARD (13.2%). All graders found that ChatGPT outperformed BARD in question categories of postoperative healing, medical eye conditions, and medications. Categorizing questions by anatomy, ChatGPT excelled in answering lacrimal questions (83.8%), while BARD performed best in the eyelid group (60.4%). ChatGPT’s answers were longer and potentially more challenging to comprehend than BARD’s. Conclusion: This study emphasizes the promising role of artificial intelligence-powered chatbots in oculoplastic patient education and support. With continued development, these chatbots may potentially assist physicians and offer patients accurate information, ultimately contributing to improved patient care while alleviating surgeon burnout. However, it is crucial to highlight that artificial intelligence may be good at answering questions, but physician oversight remains essential to ensure the highest standard of care and address complex medical cases.

Artificial Intelligence in Postoperative Care: Assessing Large Language Models for Patient Recommendations in Plastic Surgery

“…However, there was no significant difference in the accuracy among the models. This was the opposite in Al-Sharif et al [28] and in Abi-Rafeh et al [2], where GPT-3.5 outperformed Bard in providing comprehensive, accurate responses.…”

“…Conversely, Gemini proved to be significantly superior in terms of readability, with responses at an average reading level of 10th grade and an average FRE score 10 points higher than that of the other models. Al-Sharif et al [ 28 ] identified similar results, with GPT having a higher analytical reading inventory (ARI) score than BARD, indicating that a higher level of education was required to understand its responses. Perhaps providing tailored prompts requesting a specific reading level might consistently improve the readability of the answers.…”

“…In plastic surgery, several studies have analyzed LLMs’ capabilities for answering patient questions in the pre- and postoperative period for breast surgery [ 2 , 3 , 8 ], blepharoplasty [ 14 ], rhinoplasty [ 15 ], and oculoplastic surgery [ 28 ]. This underscores their potential as extremely helpful and valuable adjunctive tools for patient management and the importance of exploring their capabilities, with further visualization toward independent tools ( Figure 5 ).…”

Large Language Models for Intraoperative Decision Support in Plastic Surgery: A Comparison between ChatGPT-4 and Gemini

“…In a previous study, ChatGPT’s FKGL and FRE score indicated a hard reading level appropriate for only 33% of adults and those with a college education [ 36 ]. Furthermore, the texts produced by ChatGPT were harder than those from Bard, Gemini’s predecessor [ 37 ]. Our results show similar characteristics, as Gemini’s FKGL was significantly lower than ChatGPT-4’s, and although there was no significant difference in the FRE score, Gemini’s was higher than ChatGPT-4’s, indicating that Gemini’s responses were easier to read.…”

Artificial Empathy: User Experiences with Emotionally Intelligent Chatbots

“…Despite considerable advances in technology, including natural language processing and emotional analysis, the capability of chatbots to fully comprehend and appropriately respond to the wide array of human emotions remains in question. This limitation is echoed in the works of Rahmanti et al (2022), Vannacci (2023, and Al-Sharif (2024), who critically examine the authenticity of empathetic responses generated by AI (Al-Sharif, 2024;Rahmanti et al, 2022;Vannacci, 2023), suggesting that achieving true emotional understanding may require further technological and conceptual advancements.…”