Comparison of ChatGPT vs. Bard to Anesthesia-related Queries

Abstract

We investigated the ability of large language models (LLMs) to answer anesthesia related queries prior to surgery from a patient′s point of view. In the study, we introduced textual data evaluation metrics, investigated ″hallucinations″ phenomenon, and evaluated feasibility of using LLMs at the patient-clinician interface. ChatGPT was found to be lengthier, intellectual, and effective in its response as compared to Bard. Upon clinical evaluation, no ″hallucination″ errors were reported from ChatGPT, whereas we observed a 30.3% error in response from Bard. ChatGPT responses were difficult to read (college level difficulty) while Bard responses were more conversational and about 8th grade level from readability calculations. Linguistic quality of ChatGPT was found to be 19.7% greater for Bard (66.16 ± 13.42 vs. 55.27 ± 11.76; p=0.0037) and was independent of response length. Computational sentiment analysis revelated that polarity scores of on a Bard was significantly greater than ChatGPT (mean 0.16 vs. 0.11 on scale of -1 (negative) to 1 (positive); p=0.0323) and can be classified as ″positive″′ whereas subjectivity scores were similar across LLM′s (mean 0.54 vs 0.50 on a scale of 0 (objective) to 1 (subjective), p=0.3030). Even though the majority of the LLM responses were appropriate, at this stage these chatbots should be considered as a versatile clinical resource to assist communication between clinicians and patients, and not a replacement of essential pre-anesthesia consultation. Further efforts are needed to incorporate health literacy that will improve patient-clinical communications and ultimately, post-operative patient outcomes.

Evaluating human resources management literacy: A performance analysis of ChatGPT and bard

“…Both chatbots provided illogical answers and did not always address the knowledge content in the questions. Similarly, Patnaik & Hoffmann, U [ 53 ]. in Texas compare the performance of ChatGPT vs. Bard to answer anesthesia-related queries prior to surgery from a patient's point of view and conclude that though both gave correct responses, they should be considered as useful clinical resource to assist communication between clinicians and patients and not a replacement for the pre-anesthesia consultation.…”

“…Rao et al [ 51 ] proposed a qualitative framework for the Myers–Briggs Type Indicator [ 52 ] by comparing the performance of LLMs in evaluating human personalities. Patnaik & Hoffmann [ 53 ], who investigated the hallucinations of LLMs in clinical medicine - on patients' view of anesthesia before surgery, found that ChatGPT exhibited intellectually superior performance over Bard in psychiatry. Patil et al [ 54 ] compare the radiology knowledge of ChatGPT and Bard and conclude that both display reasonable radiology knowledge and should be used with conscious knowledge of their limitations.…”

“…In a study by Patnaik and Hoffmann [ 53 ], the readability of the responses of ChatGPT and Bard was compared using the Flesch Readability Ease score and the Flesch-Kincaid Grade Level score. The study found that the responses of Bard had a higher Flesch Readability Ease score and a lower Flesch-Kincaid Grade Level score than the responses of ChatGPT.…”