Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study

Ben-Shabat, Niv; Sharvit, Gal; Meimis, Ben; Joya, Daniel Ben; Sloma, Ariel; Kiderman, David; Shabat, Aviv; Tsur, Avishai M; Watad, Abdulla; Amital, Howard

doi:10.1016/j.ijmedinf.2022.104897

Cited by 19 publications

(10 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Clinical vignettes are a useful tool for health education, 34 assessment of healthcare professionals, 35; 36 healthcare research, 37 and symptom checker evaluation. 38 However, little is known about how clinical vignettes for health professionals are produced. 2 In 2022, large sets of vignettes were still being developed by hand, e.g.…”

Section: Discussionmentioning

confidence: 99%

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

Benoit

2023

Preprint

View full text Add to dashboard Cite

Objective To determine the capabilities of ChatGPT for rapidly generating, rewriting, and evaluating (via diagnostic and triage accuracy) sets of clinical vignettes. Design We explored the capabilities of ChatGPT for generating and rewriting vignettes. First, we gave it natural language prompts to generate 10 new sets of 10 vignettes, each set for a different common childhood illness. Next, we had it generate 10 sets of 10 vignettes given a set of symptoms from which to draw. We then had it rewrite 15 existing pediatric vignettes at different levels of health literacy. Fourth, we asked it to generate 10 vignettes written as a parent, and rewrite these vignettes as a physician, then at a grade 8 reading level, before rewriting them from the original parent's perspective. Finally, we evaluated ChatGPT for diagnosis and triage for 45 clinical vignettes previously used for evaluating symptom checkers. Setting and participants ChatGPT, a publicly available, free chatbot. Main outcome measures Our main outcomes for de novo vignette generation were whether ChatGPT followed vignette creation instructions consistently, correctly, and listed reasonable symptoms for the disease being described. For generating vignettes from pre-existing symptom sets, we examined whether the symptom sets were used without introducing extra symptoms. Our main outcome for rewriting existing standardized vignettes to match patient demographics, and rewriting vignettes between styles, was whether symptoms were dropped or added outside the original vignette. Finally, our main outcomes examining diagnostic and triage accuracy on 45 standardized patient vignettes were whether the correct diagnosis was listed first, and if the correct triage recommendation was made. Results ChatGPT was able to quickly produce varied contexts and symptom profiles when writing vignettes based on an illness name, but overused some core disease symptoms. It was able to use given symptom lists as the basis for vignettes consistently, adding one additional (though appropriate) symptom from outside the list for one disease. Pediatric vignettes rewritten at different levels of health literacy showed more complex symptoms being dropped when writing at low health literacy in 87.5% of cases. While writing at high health literacy, it added a diagnosis to 80% of vignettes (91.7% correctly diagnosed). Symptoms were retained in 90% of cases when rewriting vignettes between viewpoints. When presented with 45 vignettes, ChatGPT identified illnesses with 75.6% (95% CI, 62.6% to 88.5%) first-pass diagnostic accuracy and 57.8% (95% CI, 42.9% to 72.7%) triage accuracy. Its use does require monitoring and has caveats, which we discuss. Conclusions ChatGPT was capable, with caveats and appropriate review, of generating, rewriting, and evaluating clinical vignettes.

show abstract

Section: Discussionmentioning

confidence: 99%

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

Benoit

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…However, if chatbots are not reliable or accurate, they could misinform patients, leading to incorrect diagnoses or treatments. For instance, Ben-Shabat et al [ 41 ] evaluated the data-gathering function of eight chatbot symptom-checkers and found that the overall recall rate for all symptom-checkers was 0.32 (2280/7112; 95% CI 0.31–0.33) for all pertinent findings. These results suggest that the data-gathering performance of currently available symptom checkers is questionable.…”

Section: Discussionmentioning

confidence: 99%

Potential of AI-Driven Chatbots in Urology: Revolutionizing Patient Care Through Artificial Intelligence

Talyshinskii,

Naik,

Hameed

et al. 2023

Curr Urol Rep

View full text Add to dashboard Cite

Purpose of Review Artificial intelligence (AI) chatbots have emerged as a potential tool to transform urology by improving patient care and physician efficiency. With an emphasis on their potential advantages and drawbacks, this literature review offers a thorough assessment of the state of AI-driven chatbots in urology today. Recent Findings The capacity of AI-driven chatbots in urology to give patients individualized and timely medical advice is one of its key advantages. Chatbots can help patients prioritize their symptoms and give advice on the best course of treatment. By automating administrative duties and offering clinical decision support, chatbots can also help healthcare providers. Before chatbots are widely used in urology, there are a few issues that need to be resolved. The precision of chatbot diagnoses and recommendations might be impacted by technical constraints like system errors and flaws. Additionally, issues regarding the security and privacy of patient data must be resolved, and chatbots must adhere to all applicable laws. Important issues that must be addressed include accuracy and dependability because any mistakes or inaccuracies could seriously harm patients. The final obstacle is resistance from patients and healthcare professionals who are hesitant to use new technology or who value in-person encounters. Summary AI-driven chatbots have the potential to significantly improve urology care and efficiency. However, it is essential to thoroughly test and ensure the accuracy of chatbots, address privacy and security concerns, and design user-friendly chatbots that can integrate into existing workflows. By exploring various scenarios and examining the current literature, this review provides an analysis of the prospects and limitations of implementing chatbots in urology.

show abstract

“…Symptom checkers use machine learning algorithms to provide personalized health recommendations based on symptoms and other user information provided [ 23 ]. For infectious diseases, studies that use symptom checkers have focused on triaging COVID-19 [ 14 , 24 ]. Ada Health is a symptom checker with class II certification under the EU medical device regulation.…”

Section: What Medical Chatbots Are Available?mentioning

confidence: 99%

“…However, when used together, Ada significantly increased the accuracy rate (87%) compared with the physician alone [ 26 ]. Despite their promise, only 2 studies have compared the performance of symptom trackers against each other, with both studies finding that the superior app was from the company that funded the study [ 24 , 27 ].…”

Section: What Medical Chatbots Are Available?mentioning

confidence: 99%

ChatGPT, GPT-4, and Other Large Language Models: The Next Revolution for Clinical Microbiology?

Egli

2023

Clinical Infectious Diseases

View full text Add to dashboard Cite

ChatGPT, GPT-4, and Bard are highly advanced natural language process (NLP)-based computer programmes (chatbots) that simulate and process human conversation in written or spoken form. Recently released by the company OpenAI, ChatGPT was trained on billions of unknown text elements (tokens) and rapidly gained wide attention for its ability to respond to questions in an articulate manner across a wide range of knowledge domains. These potentially disruptive large language model (LLM) technologies have a broad range of conceivable applications in medicine and medical microbiology. In this opinion article, I will describe how chatbot technologies work and discuss the strength and weaknesses of ChatGPT, GPT-4, and other LLMs for applications in the routine diagnostic laboratory, focussing on various use cases for the pre- to post-analytical process.

show abstract

Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study

Cited by 19 publications

References 35 publications

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

ChatGPT for Clinical Vignette Generation, Revision, and Evaluation

Potential of AI-Driven Chatbots in Urology: Revolutionizing Patient Care Through Artificial Intelligence

ChatGPT, GPT-4, and Other Large Language Models: The Next Revolution for Clinical Microbiology?

Contact Info

Product

Resources

About