Parental Actionability of Educational Materials Regarding Laryngotracheal Reconstruction

Wong, Kevin; Keefe, Katherine R.; Gilad, Amir; Li, Christopher J. Chong-Sun; Levi, Jessica R.

doi:10.1001/jamaoto.2017.0942

Cited by 1 publication

(2 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The responses were then evaluated using the Patient Education Materials Assessment Tool (PEMAT). 12 The PEMAT is a validated tool that assesses the understandability and actionability of patient education materials. In this study, the PEMAT was adapted to assess the understandability and actionability of the answers generated by ChatGPT.…”

Section: Methodsmentioning

confidence: 99%

“…The actionability of each answer was scored on a scale of 0-4, with higher scores indicating greater actionability. It is important to note that certain items from the original PEMAT scale (6,12,15,16,17,18,19,24,25) were removed from the assessment because they did not apply to the answers provided by ChatGPT because of a lack of audiovisuals as well all of the ChatGPT-3.0 responses being considered as a very short material.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Usefulness and Accuracy of Artificial Intelligence Chatbot Responses to Patient Questions for Neurosurgical Procedures

Gajjar,

Kumar,

Paliwoda

et al. 2024

Neurosurgery

View full text Add to dashboard Cite

BACKGROUND AND OBJECTIVES: The Internet has become a primary source of health information, leading patients to seek answers online before consulting health care providers. This study aims to evaluate the implementation of Chat Generative Pre-Trained Transformer (ChatGPT) in neurosurgery by assessing the accuracy and helpfulness of artificial intelligence (AI)–generated responses to common postsurgical questions. METHODS: A list of 60 commonly asked questions regarding neurosurgical procedures was developed. ChatGPT-3.0, ChatGPT-3.5, and ChatGPT-4.0 responses to these questions were recorded and graded by numerous practitioners for accuracy and helpfulness. The understandability and actionability of the answers were assessed using the Patient Education Materials Assessment Tool. Readability analysis was conducted using established scales. RESULTS: A total of 1080 responses were evaluated, equally divided among ChatGPT-3.0, 3.5, and 4.0, each contributing 360 responses. The mean helpfulness score across the 3 subsections was 3.511 ± 0.647 while the accuracy score was 4.165 ± 0.567. The Patient Education Materials Assessment Tool analysis revealed that the AI-generated responses had higher actionability scores than understandability. This indicates that the answers provided practical guidance and recommendations that patients could apply effectively. On the other hand, the mean Flesch Reading Ease score was 33.5, suggesting that the readability level of the responses was relatively complex. The Raygor Readability Estimate scores ranged within the graduate level, with an average score of the 15th grade. CONCLUSION: The artificial intelligence chatbot's responses, although factually accurate, were not rated highly beneficial, with only marginal differences in perceived helpfulness and accuracy between ChatGPT-3.0 and ChatGPT-3.5 versions. Despite this, the responses from ChatGPT-4.0 showed a notable improvement in understandability, indicating enhanced readability over earlier versions.

show abstract

Section: Methodsmentioning

confidence: 99%