The exciting potential for ChatGPT in obstetrics and gynecology

Grünebaum, Amos; Chervenak, Joseph; Pollet, Susan L.; Katz, Alvin; Chervenak, Frank A.

doi:10.1016/j.ajog.2023.03.009

Cited by 140 publications

(84 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Google Research developed and tuned their Pathways Language Model (PaLM) to the medical domain using instruction prompt tuning, resulting in Med-PaLM, which scored 85% on US medical licensing examination–style questions, suggesting that LLMs can apply medical knowledge . Chatbots have demonstrated passing examination scores in multiple specialty domains, including pathology, oncology, obstetrics and gynecology, and in ophthalmology, on the Ophthalmic Knowledge Assessment Program . Other LLM applications include simplifying radiology reports, writing discharge summaries, drafting operative notes, and facilitating palliative care discussions .…”

Section: Discussionmentioning

confidence: 99%

Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Bernstein,

Zhang,

Govil

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceLarge language models (LLMs) like ChatGPT appear capable of performing a variety of tasks, including answering patient eye care questions, but have not yet been evaluated in direct comparison with ophthalmologists. It remains unclear whether LLM-generated advice is accurate, appropriate, and safe for eye patients.ObjectiveTo evaluate the quality of ophthalmology advice generated by an LLM chatbot in comparison with ophthalmologist-written advice.Design, Setting, and ParticipantsThis cross-sectional study used deidentified data from an online medical forum, in which patient questions received responses written by American Academy of Ophthalmology (AAO)–affiliated ophthalmologists. A masked panel of 8 board-certified ophthalmologists were asked to distinguish between answers generated by the ChatGPT chatbot and human answers. Posts were dated between 2007 and 2016; data were accessed January 2023 and analysis was performed between March and May 2023.Main Outcomes and MeasuresIdentification of chatbot and human answers on a 4-point scale (likely or definitely artificial intelligence [AI] vs likely or definitely human) and evaluation of responses for presence of incorrect information, alignment with perceived consensus in the medical community, likelihood to cause harm, and extent of harm.ResultsA total of 200 pairs of user questions and answers by AAO-affiliated ophthalmologists were evaluated. The mean (SD) accuracy for distinguishing between AI and human responses was 61.3% (9.7%). Of 800 evaluations of chatbot-written answers, 168 answers (21.0%) were marked as human-written, while 517 of 800 human-written answers (64.6%) were marked as AI-written. Compared with human answers, chatbot answers were more frequently rated as probably or definitely written by AI (prevalence ratio [PR], 1.72; 95% CI, 1.52-1.93). The likelihood of chatbot answers containing incorrect or inappropriate material was comparable with human answers (PR, 0.92; 95% CI, 0.77-1.10), and did not differ from human answers in terms of likelihood of harm (PR, 0.84; 95% CI, 0.67-1.07) nor extent of harm (PR, 0.99; 95% CI, 0.80-1.22).Conclusions and RelevanceIn this cross-sectional study of human-written and AI-generated responses to 200 eye care questions from an online advice forum, a chatbot appeared capable of responding to long user-written eye health posts and largely generated appropriate responses that did not differ significantly from ophthalmologist-written responses in terms of incorrect information, likelihood of harm, extent of harm, or deviation from ophthalmologist community standards. Additional research is needed to assess patient attitudes toward LLM-augmented ophthalmologists vs fully autonomous AI content generation, to evaluate clarity and acceptability of LLM-generated answers from the patient perspective, to test the performance of LLMs in a greater variety of clinical contexts, and to determine an optimal manner of utilizing LLMs that is ethical and minimizes harm.

show abstract

Section: Discussionmentioning

confidence: 99%

Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Bernstein,

Zhang,

Govil

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

show abstract

“…The model tended to manage simple cases effectively but struggled with complex ones, suggesting difficulties in handling intricate instructions. While ChatGPT exhibited some reasoning capabilities, they remained limited 6,14 . It misunderstood the implications of the ADR concept and misjudged ADR causality.…”

Section: Discussionmentioning

confidence: 99%

“…Moreover, ChatGPT has potential applications in enhancing radiological decision-making, risk prediction for various ailments, and drug discovery processes 8,9 . Beyond its technological impact, ChatGPT offers medical information to patients and may improve personalized healthcare across various clinical practices, including endocrinology 10 , hepatology 11 , cardiology 12 , anti-infective medicine 13 , obstetrics & gynecology 14 , and neurosurgery 15 .…”

Section: Introductionmentioning

confidence: 99%

Evaluating the performance of ChatGPT in clinical pharmacy

Huang¹,

Estau²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Aim To evaluate the performance of ChatGPT in key areas of clinical pharmacy practice, including prescription review, patient medication education, adverse drug reaction (ADR) recognition, ADR causality assessment, and drug counseling. Methods Quantitative and qualitative analyses were conducted to assess the accuracy and quality of ChatGPT in comparison to those of the clinical pharmacist. Results The results indicate that ChatGPT is excellent in drug counseling and weak in prescription review, patient medication education, ADR recognition, and ADR causality. Conclusions While ChatGPT holds promise in clinical pharmacy practice as a supplementary tool, the ability of ChatGPT to handle complex problems needs further improvement and refinement.

show abstract

“…ChatGPT users were on average 19% faster in completing the questionnaire than users of other sources, although this difference was non signifi cant. ChatGPT users had greater odds of maximum scores than nonChatGPT users: ORs were 2.34 (composite), 1.96 (complete ness) and 2.89 (nuance), with 95% CIs not encompassing 1, and cor rected p values of 0.0037, 0.022, and 3.09x10 5 , respectively. The OR for accuracy was 1.33 (non significant).…”

mentioning

confidence: 95%

“…A strength of our study is that we employed a comprehensive, two layered approach that goes beyond similar studies in other specialties by number of users, number of questions asked, num bers of outcomes, and the complementary use of two methods 5 . A limitation is the potential lack of power to detect significant dif ferences in response speed between ChatGPT and non ChatGPT users.…”

mentioning

confidence: 99%

The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two‐layer assessment

Luykx,

Gerritse,

Habets

et al. 2023

World Psychiatry

View full text Add to dashboard Cite

The exciting potential for ChatGPT in obstetrics and gynecology

Cited by 140 publications

References 50 publications

Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Evaluating the performance of ChatGPT in clinical pharmacy

The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two‐layer assessment

Contact Info

Product

Resources

About