Assessing the readability and quality of online information on Bell's palsy

Raja, Haseem; Fitzpatrick, Niall

doi:10.1017/s0022215122002626

Cited by 4 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluated readability using multiple scales, including the previously validated Flesh-Kincaid grade level, Gunning Fog index, the Simple Measure of Gobbledygook score, and the Coleman-Liau index (Table 1). These measures consider sentence length, word length, and overall text complexity, and provide a US academic grade level for an average student . We calculated readability scores for each scale using an online tool…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures

Decker,

Trang,

Ramirez

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

ImportanceInformed consent is a critical component of patient care before invasive procedures, yet it is frequently inadequate. Electronic consent forms have the potential to facilitate patient comprehension if they provide information that is readable, accurate, and complete; it is not known if large language model (LLM)-based chatbots may improve informed consent documentation by generating accurate and complete information that is easily understood by patients.ObjectiveTo compare the readability, accuracy, and completeness of LLM-based chatbot- vs surgeon-generated information on the risks, benefits, and alternatives (RBAs) of common surgical procedures.Design, Setting, and ParticipantsThis cross-sectional study compared randomly selected surgeon-generated RBAs used in signed electronic consent forms at an academic referral center in San Francisco with LLM-based chatbot-generated (ChatGPT-3.5, OpenAI) RBAs for 6 surgical procedures (colectomy, coronary artery bypass graft, laparoscopic cholecystectomy, inguinal hernia repair, knee arthroplasty, and spinal fusion).Main Outcomes and MeasuresReadability was measured using previously validated scales (Flesh-Kincaid grade level, Gunning Fog index, the Simple Measure of Gobbledygook, and the Coleman-Liau index). Scores range from 0 to greater than 20 to indicate the years of education required to understand a text. Accuracy and completeness were assessed using a rubric developed with recommendations from LeapFrog, the Joint Commission, and the American College of Surgeons. Both composite and RBA subgroup scores were compared.ResultsThe total sample consisted of 36 RBAs, with 1 RBA generated by the LLM-based chatbot and 5 RBAs generated by a surgeon for each of the 6 surgical procedures. The mean (SD) readability score for the LLM-based chatbot RBAs was 12.9 (2.0) vs 15.7 (4.0) for surgeon-generated RBAs (P = .10). The mean (SD) composite completeness and accuracy score was lower for surgeons’ RBAs at 1.6 (0.5) than for LLM-based chatbot RBAs at 2.2 (0.4) (P &lt; .001). The LLM-based chatbot scores were higher than the surgeon-generated scores for descriptions of the benefits of surgery (2.3 [0.7] vs 1.4 [0.7]; P &lt; .001) and alternatives to surgery (2.7 [0.5] vs 1.4 [0.7]; P &lt; .001). There was no significant difference in chatbot vs surgeon RBA scores for risks of surgery (1.7 [0.5] vs 1.7 [0.4]; P = .38).Conclusions and RelevanceThe findings of this cross-sectional study suggest that despite not being perfect, LLM-based chatbots have the potential to enhance informed consent documentation. If an LLM were embedded in electronic health records in a manner compliant with the Health Insurance Portability and Accountability Act, it could be used to provide personalized risk information while easing documentation burden for physicians.

show abstract

Section: Methodsmentioning

confidence: 99%

“…29 These measures consider sentence length, word length, and overall text complexity, and provide a US academic grade level for an average student. 30,31 We calculated readability scores for each scale using an online tool. 32…”

Section: Measurements Of Readabilitymentioning

confidence: 99%

Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures

Decker,

Trang,

Ramirez

et al. 2023

JAMA Netw Open

View full text Add to dashboard Cite

show abstract

BPPV Information on Google Versus AI (ChatGPT)

Bellinger,

De La Chapa,

Kwak

et al. 2023

Otolaryngol.--head neck surg.

View full text Add to dashboard Cite

ObjectiveTo quantitatively compare online patient education materials found using traditional search engines (Google) versus conversational Artificial Intelligence (AI) models (ChatGPT) for benign paroxysmal positional vertigo (BPPV).Study DesignThe top 30 Google search results for “benign paroxysmal positional vertigo” were compared to the OpenAI conversational AI language model, ChatGPT, responses for 5 common patient questions posed about BPPV in February 2023. Metrics included readability, quality, understandability, and actionability.SettingOnline information.MethodsValidated online information metrics including Flesch‐Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE), DISCERN instrument score, and Patient Education Materials Assessment Tool for Printed Materials were analyzed and scored by reviewers.ResultsMean readability scores, FKGL and FRE, for the Google webpages were 10.7 ± 2.6 and 46.5 ± 14.3, respectively. ChatGPT responses had a higher FKGL score of 13.9 ± 2.5 (P < .001) and a lower FRE score of 34.9 ± 11.2 (P = .005), both corresponding to lower readability. The Google webpages had a DISCERN part 2 score of 25.4 ± 7.5 compared to the individual ChatGPT responses with a score of 17.5 ± 3.9 (P = .001), and the combined ChatGPT responses with a score of 25.0 ± 0.9 (P = .928). The average scores of the reviewers for all ChatGPT responses for accuracy were 4.19 ± 0.82 and 4.31 ± 0.67 for currency.ConclusionThe results of this study suggest that the information on ChatGPT is more difficult to read, of lower quality, and more difficult to comprehend compared to information on Google searches.

show abstract

Functional endoscopic sinus surgery: assessing the readability and quality of online information

Raja

Patel

2023

annals

View full text Add to dashboard Cite

Introduction The aim of this study was to assess the readability and quality of online information on functional endoscopic sinus surgery (FESS). Methods The term ‘functional endoscopic sinus surgery’ was entered into the Google, Bing and Yahoo search engines. The first 30 websites for each search engine were screened. Readability was assessed using the Flesch–Kincaid Reading Ease Score (FRES), Flesch–Kincaid Grade (FKG), Simple Measure of Gobbledygook (SMOG) Index, and Gunning Fog Index (GFI). Quality was assessed using the DISCERN instrument. Spearman’s correlation between quality and readability was calculated. Results Thirty-three websites met the inclusion criteria. The mean and standard deviations for the FRES, FKG, SMOG, GFI, and DISCERN scores were 49 (13.1), 10.9 (2.5), 10.2 (1.9), 13.6 (2.4), and 51.1 (12.8), respectively. A positive correlation was noted between the FRES and DISCERN (R=0.357, p=0.041). Conclusions Online information on FESS is generally written above the recommended reading levels and of fair quality.

show abstract

Assessing the readability and quality of online information on Bell's palsy

Cited by 4 publications

References 11 publications

Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures

Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures

BPPV Information on Google Versus AI (ChatGPT)

Functional endoscopic sinus surgery: assessing the readability and quality of online information

Contact Info

Product

Resources

About