Assessment of Readability, Understandability and Actionability of a Fully AI-generated Patient Information Brochure, a Cross-sectional Study (Preprint)

Abstract

BACKGROUND A gap exists between the readability of readily available patient education materials and the literacy skill of the average US adult. With the current advancements in artificial intelligence (AI), AI tools could provide the solution. OBJECTIVE To evaluate the readability, understandability and actionability of patient information brochures fully generated by artificial intelligence. METHODS Patient information brochures about the three most common surgical pathologies (appendicitis, cholecystitis, inguinal hernia) were generated by ChatGPT-4 and Bing Image Creator and subsequently evaluated for readability using readability scores and for understandability and actionability by surgeons using Patient Educational Materials Assessment for Printable materials (PEMAT-P). RESULTS Brochures generated by ChatGPT and DALL-E scored well on text understandability (93.6%) but poorly on visual aids (18.67%) and actionability (64%). The mean Flesch reading ease was 41.54, Flesch-Kincaid grade level: 10.52, Gunning Fog Index: 11.98, SMOG Index: 12.78, Coleman Liau Index: 13.3 and Automated Readability Index: 9.98. CONCLUSIONS Patient information brochures generated by ChatGPT and Bing Image Creator had high text understandability, low visual aids understandability and low actionability. The readability levels of the generated content exceed the average literacy level of the US adult population.