The Effects of a Health Care Chatbot’s Complexity and Persona on User Trust, Perceived Usability, and Effectiveness: Mixed Methods Study

Abstract

Background The rising adoption of telehealth provides new opportunities for more effective and equitable health care information mediums. The ability of chatbots to provide a conversational, personal, and comprehendible avenue for learning about health care information make them a promising tool for addressing health care inequity as health care trends continue toward web-based and remote processes. Although chatbots have been studied in the health care domain for their efficacy for smoking cessation, diet recommendation, and other assistive applications, few studies have examined how specific design characteristics influence the effectiveness of chatbots in providing health information. Objective Our objective was to investigate the influence of different design considerations on the effectiveness of an educational health care chatbot. Methods A 2×3 between-subjects study was performed with 2 independent variables: a chatbot’s complexity of responses (eg, technical or nontechnical language) and the presented qualifications of the chatbot’s persona (eg, doctor, nurse, or nursing student). Regression models were used to evaluate the impact of these variables on 3 outcome measures: effectiveness, usability, and trust. A qualitative transcript review was also done to review how participants engaged with the chatbot. Results Analysis of 71 participants found that participants who received technical language responses were significantly more likely to be in the high effectiveness group, which had higher improvements in test scores (odds ratio [OR] 2.73, 95% CI 1.05-7.41; P=.04). Participants with higher health literacy (OR 2.04, 95% CI 1.11-4.00, P=.03) were significantly more likely to trust the chatbot. The participants engaged with the chatbot in a variety of ways, with some taking a conversational approach and others treating the chatbot more like a search engine. Conclusions Given their increasing popularity, it is vital that we consider how chatbots are designed and implemented. This study showed that factors such as chatbots’ persona and language complexity are two design considerations that influence the ability of chatbots to successfully provide health care information.

ChatGPT Review: A Sophisticated Chatbot Models in Medical & Health-related Teaching and Learning

“…Another study found that a chatbot's complexity of replies and persona credentials affect its health care information delivery. Technical language responses from chatbot replies were found to be effective and boost test results, while high health literacy groups are more likely to trust chatbots [65]. ChatGPT has several healthcare and medical applications.…”

Ethical Implications of Chatbot Utilization in Nephrology

“…While chatbots can provide valuable information and support, they may also affect the traditional dynamics of the patient–provider relationship [ 47 ]. Patients may perceive chatbots as substitutes for human interaction, potentially influencing their satisfaction, trust, and overall healthcare experience [ 48 , 49 ] ( Figure 5 and Figure 6 ).…”

“…While chatbots can provide valuable information and support, they may also affect the traditional dynamics of the patient-provider relationship [47]. Patients may perceive chatbots as substitutes for human interaction, potentially influencing their satisfaction, trust, and overall healthcare experience [48,49] (Figures 5 and 6). The utilization of chatbots in nephrology not only impacts clinical decision-making but also raises ethical considerations regarding the patient-provider relationship and communication [45].…”

“…While chatbots can provide valuable information and support, they may also affect the traditional dynamics of the patient-provider relationship [47]. Patients may perceive chatbots as substitutes for human interaction, potentially influencing their satisfaction, trust, and overall healthcare experience [48,49] (Figures 5 and 6). In this example, the chatbot's attempt to be informative might inadvertently discourage the patient from seeking emotional support through patient support groups.…”