Quantitative User Data From a Chatbot Developed for Women With Gestational Diabetes Mellitus: Observational Study

Abstract

Background The rising prevalence of gestational diabetes mellitus (GDM) calls for the use of innovative methods to inform and empower these pregnant women. An information chatbot, Dina, was developed for women with GDM and is Norway’s first health chatbot, integrated into the national digital health platform. Objective The aim of this study is to investigate what kind of information users seek in a health chatbot providing support on GDM. Furthermore, we sought to explore when and how the chatbot is used by time of day and the number of questions in each dialogue and to categorize the questions the chatbot was unable to answer (fallback). The overall goal is to explore quantitative user data in the chatbot’s log, thereby contributing to further development of the chatbot. Methods An observational study was designed. We used quantitative anonymous data (dialogues) from the chatbot’s log and platform during an 8-week period in 2018 and a 12-week period in 2019 and 2020. Dialogues between the user and the chatbot were the unit of analysis. Questions from the users were categorized by theme. The time of day the dialogue occurred and the number of questions in each dialogue were registered, and questions resulting in a fallback message were identified. Results are presented using descriptive statistics. Results We identified 610 dialogues with a total of 2838 questions during the 20 weeks of data collection. Questions regarding blood glucose, GDM, diet, and physical activity represented 58.81% (1669/2838) of all questions. In total, 58.0% (354/610) of dialogues occurred during daytime (8 AM to 3:59 PM), Monday through Friday. Most dialogues were short, containing 1-3 questions (340/610, 55.7%), and there was a decrease in dialogues containing 4-6 questions in the second period (P=.013). The chatbot was able to answer 88.51% (2512/2838) of all posed questions. The mean number of dialogues per week was 36 in the first period and 26.83 in the second period. Conclusions Frequently asked questions seem to mirror the cornerstones of GDM treatment and may indicate that the chatbot is used to quickly access information already provided for them by the health care service but providing a low-threshold way to access that information. Our results underline the need to actively promote and integrate the chatbot into antenatal care as well as the importance of continuous content improvement in order to provide relevant information.

ChatGPT- versus human-generated answers to frequently asked questions about diabetes: a Turing test-inspired survey among employees of a Danish diabetes center

“…Based on a 20-week data collection, the authors reported that the chatbot could answer 89% of almost 3,000 questions, providing women with readily accessible information about their condition. 12 Integration of such applications in national digital health platforms governed by the public sector might increase public trust needed for successful implementation. Considering the dominance of the private sector in the field of AI, and that people are more willing to share their data for AI research with the public sector than with private companies, 13 it is relevant to make partnerships between different actors and stakeholders, and patient organizations.…”

“…In Norway, the first chatbot integrated into the national health service platform was developed to inform and empower women with gestational diabetes. 12 Our previous work suggests that the majority of patients are open to AI-based solutions, with higher acceptance rates in low-risk situations. 13 This calls for studies investigating the knowledge and value of large language models, such as ChatGPT, in everyday, practical scenarios and routine clinical care.…”

Insights of Parents and Parents‐To‐Be in Using Chatbots to Improve Their Preconception, Pregnancy, and Postpartum Health: A Mixed Studies Review

“…15,30,37,[39][40][41]43,45 Overall, participants were satisfied and felt that chatbots were able to deliver "trustworthy information" 31 and answer the majority of their questions. [14][15][16]30,43 Although participants knew that the chatbot was computer generated, some treated it like their "mum," health care professional, friend, or confidante and recounted personal anecdotes while interacting with the chatbot. 14,31,42 Host of Health Benefits Chatbots were able to improve participants' health outcomes holistically.…”

“…15,30,40,41,43,45 Lastly, chatbots empowered women with relevant knowledge and enabled them to make their own health care decisions. 14,16,38…”

“…Ongoing feedback is needed to assess the content and adequacy of information provided by chatbots. 16 Researchers and chatbot developers can track and analyze users' chat histories and feedback and upgrade the chatbot to better align with users' needs and requests. 16,31 Targeted users could also be invited to participate during the chatbot development stages and take part in acceptability and function testing.…”