Assessing the Reliability of ChatGPT Chatbots for Smoking Cessation: A Content Analysis (Preprint)

Abstract

BACKGROUND Large Language Model (LLM) AI chatbots using generative language can offer smoking cessation information and advice. However, little is known about the reliability of information provided to users. OBJECTIVE This study aims to examine whether 3 ChatGPT chatbots – the World Health Organization’s (WHO) Sarah, BeFreeGPT, and BasicGPT – provide reliable information on how to quit smoking. METHODS A list of quit smoking queries was generated from frequent quit smoking searches on Google related to “how to quit smoking” (N=12). Each query was given to each chatbot, and responses were analyzed for their adherences to an index developed from the United States Preventive Services Task Force (USPSTF) public health guidelines for quitting smoking and counseling principles. Responses were independently coded by 2 reviewers and differences resolved by a third coder. RESULTS Across chatbots and queries, chatbot responses were rated as being adherent to 57.1% of the items on the adherence index. Sarah’s adherence (72.2%) was significantly higher than BeFreeGPT (50.0%) and BasicGPT (47.8%) (p<.01). The majority of Chatbot responses had clear language (97.3%) and included a recommendation to seek out professional counseling (80.3%). About half of responses included the recommendation to consider using nicotine replacement therapy (NRT) (52.7%), the recommendation to seek out social support from friends and family (55.6%), and information on how to deal with cravings when quitting smoking (44.4%). Least common was information about considering the use of non-NRT prescription drugs (14.1%). Finally, some type of misinformation was present in 22.0% of responses. Specific queries that were most challenging for the chatbots included queries on “how to quit smoking cold turkey,” “… with vapes,” “…with gummies,” “…with a necklace,” and “…with hypnosis.” All chatbots showed resilience to adversarial attacks that were intended to derail the conversation. CONCLUSIONS LLM chatbots varied in their adherence to quit smoking guidelines and counseling principles. While chatbots reliably provided some types of information, they omitted other types, as well as occasionally provided misinformation, especially for queries about less evidence-based methods of quitting. LLM chatbot instructions can be revised to compensate for these weaknesses. CLINICALTRIAL n/a