Quality of information and appropriateness of ChatGPT outputs for urology patients

Cocci, Andrea; Pezzoli, Marta; Re, Mattia Lo; Russo, Giorgio Ivan; Asmundo, Maria Giovanna; Fode, Mikkel; Cacciamani, Giovanni; Cimino, Silvia; Minervini, Andrea; Durukan, Emil

doi:10.1038/s41391-023-00705-y

Cited by 46 publications

(18 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Readability calculation (Table 1) utilized the following validated formulae: Flesch-Kincaid Reading Ease Score, Flesch-Kincaid Grade Level, Gunning-Fog Score, Simple Measure of Gobbledygook, Coleman-Liau Index, and Automated Readability Index. 1,20,21…”

Section: Methodsmentioning

confidence: 99%

“…Two blinded urologists scored responses for accuracy (“Is the response evidence-based and medically accurate?”), comprehensiveness ("Does the response provide sufficient information to fully inform patients about their diagnosis/treatment?”), and understandability (“Can the response be easily understood by average patients?”) using Likert items (1 = Poor, 2 = Needs Improvement, 3 = Fair, 4 = Good, 5 = Excellent). 1,20-24 Accuracy was assessed using AUA guidelines. Comprehensiveness and understandability were scored qualitatively.…”

Section: Methodsmentioning

confidence: 99%

“…Patients increasingly go online for health advice, particularly for sensitive issues like sexual health. 1,2 As technology companies continue to innovate artificial intelligence (AI) applications, this trend will increase. 3-5 Modern large language models, namely OpenAI’s ChatGPT, allow free text entry and generate novel responses.…”

mentioning

confidence: 99%

“…ChatGPT can perform diverse physician-oriented tasks, including writing notes, improving empathy in written communications, 1,9 completing certification exams, 10 and simplifying scientific literature. 11 However, there is limited understanding of AI use by patients, who have recently begun turning to new platforms for their health information.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health

Shah,

Ghosh,

Hochberg

et al. 2024

Urology Practice

View full text Add to dashboard Cite

Introduction:ChatGPT is an artificial intelligence platform available to patients seeking medical advice. Traditionally, urology patients consulted official provider-created materials, particularly the Urology Care FoundationÔ (UCF). Today, men increasingly go online due to the rising costs of health care and the stigma surrounding sexual health. Online health information is largely inaccessible to laypersons as it exceeds the recommended American sixth to eighth grade reading level. We conducted a comparative assessment of patient education materials generated by ChatGPT vs UCF regarding men's health conditions.Methods: All 6 UCF men's health resources were identified. ChatGPT responses were generated using patient questions obtained from UCF. Adjusted ChatGPT responses were generated by prompting, "Explain it to me like I am in sixth grade." Textual analysis was performed using sentence, word, syllable, and complex word count. Six validated formulae were used for readability analysis. Two physicians independently scored responses for accuracy, comprehensiveness, and understandability. Statistical analysis involved Wilcoxon matched-pairs test.Results: ChatGPT responses were longer and more complex. Both UCF and ChatGPT failed official readability standards, although ChatGPT performed significantly worse across all 6 topics (all P < .001). Conversely, adjusted ChatGPT readability typically surpassed UCF, even meeting the recommended level for 2 topics. Qualitatively, UCF and ChatGPT had comparable accuracy, although ChatGPT had better comprehensiveness and worse understandability.Conclusions: When comparing readability, ChatGPT-generated education is less accessible than provider-written content, although neither meets the recommended level. Our analysis indicates that specific artificial intelligence prompts can simplify educational materials to meet national standards and accommodate individual literacy.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health

Shah,

Ghosh,

Hochberg

et al. 2024

Urology Practice

View full text Add to dashboard Cite

show abstract

“…1 Our findings highlighted that this technology is capable of providing medical informationdeven if not fully comprehensive and correctdconfirming the findings of other studies in urology. 2,3 The risk is that in future this technology could potentially replace the more commonly used "Dr Google" 4 to gather health care information. The human-like interaction that artificial intelligence (AI)epowered chatbots provide could inspire trust in patients, and therefore be widely used for a range of different health care questions, from diagnosis to treatments.…”

Section: Evaluating the Effectiveness Of Artificialmentioning

confidence: 99%

Evaluating the Effectiveness of Artificial Intelligence–powered Large Language Models Application in Disseminating Appropriate and Readable Health Information in Urology. Letter.

Di,

Wen

2023

Journal of Urology

View full text Add to dashboard Cite

Large Language Model‐Based Chatbots in Higher Education

Yigci,

Eryilmaz,

Yetisen

et al. 2024

Advanced Intelligent Systems

View full text Add to dashboard Cite

Large language models (LLMs) are artificial intelligence (AI) platforms capable of analyzing and mimicking natural language processing. Leveraging deep learning, LLM capabilities have been advanced significantly, giving rise to generative chatbots such as Generative Pre‐trained Transformer (GPT). GPT‐1 was initially released by OpenAI in 2018. ChatGPT's release in 2022 marked a global record of speed in technology uptake, attracting more than 100 million users in two months. Consequently, the utility of LLMs in fields including engineering, healthcare, and education has been explored. The potential of LLM‐based chatbots in higher education has sparked significant interest and ignited debates. LLMs can offer personalized learning experiences and advance asynchronized learning, potentially revolutionizing higher education, but can also undermine academic integrity. Although concerns regarding AI‐generated output accuracy, the spread of misinformation, propagation of biases, and other legal and ethical issues have not been fully addressed yet, several strategies have been implemented to mitigate these limitations. Here, the development of LLMs, properties of LLM‐based chatbots, and potential applications of LLM‐based chatbots in higher education are discussed. Current challenges and concerns associated with AI‐based learning platforms are outlined. The potentials of LLM‐based chatbot use in the context of learning experiences in higher education settings are explored.

show abstract

Quality of information and appropriateness of ChatGPT outputs for urology patients

Cited by 46 publications

References 12 publications

Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health

Comparison of ChatGPT and Traditional Patient Education Materials for Men’s Health

Evaluating the Effectiveness of Artificial Intelligence–powered Large Language Models Application in Disseminating Appropriate and Readable Health Information in Urology. Letter.

Large Language Model‐Based Chatbots in Higher Education

Contact Info

Product

Resources

About