Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions

Ayoub, Noel; Lee, Yujin; Grimm, David R.; Balakrishnan, Karthik

doi:10.1001/jamaoto.2023.0704

Cited by 46 publications

(20 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While Ayoub et al were able to bypass this restriction by telling the ChatGPT model to respond using a fifth grade reading level, it was our intention to utilize ChatGPT similar to how a layperson would. 10 Intricate knowledge of how to interact with AI is not commonplace, so we sought to emulate a realistic environment of one using ChatGPT similar to how a search engine would be used. As…”

Section: Discussionmentioning

confidence: 99%

“…as ChatGPT in its present condition cannot reference resources. 10 This is of significant concern specifically for ChatGPT-3. A final observation was that the lay public may have unrealistic biases or exaggerated expectations of AI.…”

Section: Discussionmentioning

confidence: 99%

“…Additionally, the average layperson may be left feeling threatened or scared by the complex terminology and medically‐oriented feedback (Figure 2). ChatGPT prefaced each of its responses by saying its information should not be taken as medical advice, a shortcoming that should be heeded by clinicians and patients alike 10 . Medical information from LLM should be acknowledged similar to a reference, something that points us in the direction of primary information, rather than the material itself.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Can ChatGPT help patients answer their otolaryngology questions?

Zalzal,

Abraham,

Cheng

et al. 2023

Laryngoscope Investig Oto

View full text Add to dashboard Cite

BackgroundOver the past year, the world has been captivated by the potential of artificial intelligence (AI). The appetite for AI in science, specifically healthcare is huge. It is imperative to understand the credibility of large language models in assisting the public in medical queries.ObjectiveTo evaluate the ability of ChatGPT to provide reasonably accurate answers to public queries within the domain of Otolaryngology.MethodsTwo board‐certified otolaryngologists (HZ, RS) inputted 30 text‐based patient queries into the ChatGPT‐3.5 model. ChatGPT responses were rated by physicians on a scale (accurate, partially accurate, incorrect), while a similar 3‐point scale involving confidence was given to layperson reviewers. Demographic data involving gender and education level was recorded for the public reviewers. Inter‐rater agreement percentage was based on binomial distribution for calculating the 95% confidence intervals and performing significance tests. Statistical significance was defined as p < .05 for two‐sided tests.ResultsIn testing patient queries, both Otolaryngology physicians found that ChatGPT answered 98.3% of questions correctly, but only 79.8% (range 51.7%–100%) of patients were confident that the AI model was accurate in its responses (corrected agreement = 0.682; p < .001). Among the layperson responses, the corrected coefficient was of moderate agreement (0.571; p < .001). No correlation was noted among age, gender, or education level for the layperson responses.ConclusionChatGPT is highly accurate in responding to questions posed by the public with regards to Otolaryngology from a physician standpoint. Public reviewers were not fully confident in believing the AI model, with subjective concerns related to less trust in AI answers compared to physician explanation. Larger evaluations with a representative public sample and broader medical questions should immediately be conducted by appropriate organizations, governing bodies, and/or governmental agencies to instill public confidence in AI and ChatGPT as a medical resource.Level of Evidence4.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Can ChatGPT help patients answer their otolaryngology questions?

Zalzal,

Abraham,

Cheng

et al. 2023

Laryngoscope Investig Oto

View full text Add to dashboard Cite

show abstract

“…It is also able to provide high‐quality answers to patient questions in pre‐ and postoperative care in oral and maxillofacial surgery, though the author suggests it be used in conjunction with surgeon experience 13 . Within otolaryngology, ChatGPT provides patients with procedure‐specific instructions equivalent to institutional standards 14 . However, ChatGPT instructions were equal to and inferior to Google's in terms of understandability and actionability, respectively 14 .…”

Section: Discussionmentioning

confidence: 99%

“…Within otolaryngology, ChatGPT provides patients with procedure‐specific instructions equivalent to institutional standards 14 . However, ChatGPT instructions were equal to and inferior to Google's in terms of understandability and actionability, respectively 14 . Patients are likely to find ChatGPT to be a useful and accessible resource.…”

Section: Discussionmentioning

confidence: 99%

Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios

Qu,

Qureshi,

Petersen

et al. 2023

OTO Open

View full text Add to dashboard Cite

ObjectiveTo evaluate the clinical applications and limitations of chat generative pretrained transformer (ChatGPT) in otolaryngology.Study DesignCross‐sectional survey.SettingTertiary academic center.MethodsChatGPT 4.0 was queried for diagnoses and management plans for 20 physician‐written clinical vignettes in otolaryngology. Attending physicians were then asked to rate the difficulty of the clinical vignettes and agreement with the differential diagnoses and management plans of ChatGPT responses on a 5‐point Likert scale. Summary statistics were calculated. Univariate ordinal regression was then performed between vignette difficulty and quality of the diagnoses and management plans.ResultsEleven attending physicians completed the survey (61% response rate). Overall, vignettes were rated as very easy to neutral difficulty (range of median score: 1.00‐4.00; overall median 2.00). There was a high agreement with the differential diagnosis provided by ChatGPT (range of median score: 3.00‐5.00; overall median: 5.00). There was also high agreement with treatment plans (range of median score: 3.00‐5.00; overall median: 5.00). There was no association between vignette difficulty and agreement with differential diagnosis or treatment. Lower diagnosis scores had greater odds of having lower treatment scores.ConclusionGenerative artificial intelligence models like ChatGPT are being rapidly adopted in medicine. Performance with curated, easy‐to‐moderate difficulty otolaryngology scenarios indicate high agreement with physicians for diagnosis and management. However, a decreased quality in diagnosis is associated with decreased quality in management. Further research is necessary on ChatGPT's ability to handle unstructured clinical information.

show abstract

JAMA Otolaryngology–Head & Neck Surgery—The Year in Review, 2023

Piccirillo

2024

JAMA Otolaryngol Head Neck Surg

View full text Add to dashboard Cite

Brought new challenges and opportunities to medical publishing, JAMA, and the JAMA Network. In January, JAMA and the JAMA Network adopted a new policy for public access to scientific research findings that permits authors of original research to deposit their accepted manuscript in a public repository on the day that the manuscript is published by the JAMA Network. 1 This new policy allows authors to comply with existing funder mandates for open access and, in combination with existing policy for data sharing, 2 fulfills National Institutes of Health requirements. In a widely viewed Editorial, changes to the JAMA Network Instructions to Authors for the use of nonhuman artificial intelligence (AI) were described, including that AI tools cannot be named as authors. 3 In November 2022, OpenAI released a major update to the large language model (LLM) ChatGPT. Quickly thereafter several examples of the use of LLMs with ChatGPT listed as an author were described. 4 To address the potential misuse of AI in publication, the author responsibilities for reporting the use of AI to create or edit content, including images, were updated. 5 At the time of submission, authors are now required to answer the following question to indicate whether AI tools were used in the creation of the manuscript: "Did you use AI, a language model, machine learning, or similar technologies to create or assist with creation or editing of any of the content in this submission." If the answer is yes, authors are required to provide a description of the AI-generated content as well as information about the AI tool or model and to confirm that they take responsibility for the integrity of the content generated by these tools. The goal of these changes is to ensure the responsible use of AI language models and transparent reporting of how these tools are used.This past year, we were quite busy at the journal. The number of submissions remained at historically high levels. I thank all the authors who submitted their work to the journal for consideration for publication. I especially thank the editors, members of the editorial board, peer reviewers, 6 and the outstanding JAMA Network publication staff for their continued hard work and dedication to the journal.The journal received 1539 manuscripts overall and 1182 research manuscripts and published 216 (14%) manuscripts overall and 97 (11%) Original Investigations , including 7 clinical trials and 13 (17%) Research Letters. The primary subject areas of published manuscripts represents the broad diversity of our specialty: oncology (29%), otology and neurotology (14%), general otolaryngology (11%), pediatric otolaryngology (11%), laryngology (6%), rhinology (5%), sleep medicine (3%), equity, diversity, and inclusion (3%), and endocrinology (2%). Despite the large number of submissions, the journal was able to complete manuscript reviews from receipt to first decision with peer Receipt to first decision without peer review, median, d 8Receipt to first decision with peer review, median, d 53Peer reviewer ...

show abstract

Comparison Between ChatGPT and Google Search as Sources of Postoperative Patient Instructions

Abstract: This qualitative study rates the level of understandability, actionability, and procedure-specific content in postoperative instructions generated from ChatGPT, Google Search, and Stanford University.

Cited by 46 publications

References 5 publications

Can ChatGPT help patients answer their otolaryngology questions?

Can ChatGPT help patients answer their otolaryngology questions?

Diagnostic and Management Applications of ChatGPT in Structured Otolaryngology Clinical Scenarios

JAMA Otolaryngology–Head & Neck Surgery—The Year in Review, 2023

Contact Info

Product

Resources

About