Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study

Abstract

Background The potential of chatbots for screening and monitoring COVID-19 was envisioned since the outbreak of the disease. Chatbots can help disseminate up-to-date and trustworthy information, promote healthy social behavior, and support the provision of health care services safely and at scale. In this scenario and in view of its far-reaching postpandemic impact, it is important to evaluate user experience with this kind of application. Objective We aimed to evaluate the quality of user experience with a COVID-19 chatbot designed by a large telehealth service in Brazil, focusing on the usability of real users and the exploration of strengths and shortcomings of the chatbot, as revealed in reports by participants in simulated scenarios. Methods We examined a chatbot developed by a multidisciplinary team and used it as a component within the workflow of a local public health care service. The chatbot had 2 core functionalities: assisting web-based screening of COVID-19 symptom severity and providing evidence-based information to the population. From October 2020 to January 2021, we conducted a mixed methods approach and performed a 2-fold evaluation of user experience with our chatbot by following 2 methods: a posttask usability Likert-scale survey presented to all users after concluding their interaction with the bot and an interview with volunteer participants who engaged in a simulated interaction with the bot guided by the interviewer. Results Usability assessment with 63 users revealed very good scores for chatbot usefulness (4.57), likelihood of being recommended (4.48), ease of use (4.44), and user satisfaction (4.38). Interviews with 15 volunteers provided insights into the strengths and shortcomings of our bot. Comments on the positive aspects and problems reported by users were analyzed in terms of recurrent themes. We identified 6 positive aspects and 15 issues organized in 2 categories: usability of the chatbot and health support offered by it, the former referring to usability of the chatbot and how users can interact with it and the latter referring to the chatbot’s goal in supporting people during the pandemic through the screening process and education to users through informative content. We found 6 themes accounting for what people liked most about our chatbot and why they found it useful—3 themes pertaining to the usability domain and 3 themes regarding health support. Our findings also identified 15 types of problems producing a negative impact on users—10 of them related to the usability of the chatbot and 5 related to the health support it provides. Conclusions Our results indicate that users had an overall positive experience with the chatbot and found the health support relevant. Nonetheless, qualitative evaluation of the chatbot indicated challenges and directions to be pursued in improving not only our COVID-19 chatbot but also health chatbots in general.

Post-COVID-19 cognitive symptoms in patients assisted by a teleassistance service: a retrospective cohort study

“…For the UFMG students and staff, access to the TeleCOVID-MG service was done through an online symptom auto-verification application developed by the pandemic committee of the university. Upon identification of any flu-like symptoms, patients were referred to a chatbot, a computer program that collected name, Brazilian identification number, telephone number, warning signs or any comorbidities which increased the risk of worse outcomes ( 11 ). In cases of suspicion of flu-like syndrome and according to the severity of the symptoms, the patient was assisted by a nurse or a physician through a phone call teleconsultation.…”

Eficácia Da Utilização De Chatbots Na Triagem De Sintomas De Covid-19: Uma Revisão Integrativa

“…A partir disso, foi realizada a leitura na íntegra dos artigos restantes, sendo excluídos aqueles que não respondiam a pergunta norteadora. (CHAGAS et al, 2023;WANG et al, 2022) Contudo, foram frisadas também possíveis melhorias na construção do sistema, especialmente no que diz respeito a usabilidade (como exemplo, pode-se citar o fluxo da conversa, em que o usuário não poderia voltar a mesma pergunta caso a respondesse erroneamente) e ao suporte em saúde (como o baixo detalhamento de informações em saúde, ou a falta de orientação de instruções práticas de conduta após a triagem) (CHAGAS et al, 2023;WANG et al, 2022).…”

“…No entanto, é importante reconhecer que o uso de chatbots pode ser limitado em países com poucos recursos e alta disparidade socioeconômica, devido à necessidade de acesso a dispositivos com internet e alfabetização para a leitura e interpretação do texto fornecido pelo chatbot (CHAGAS et al, 2023;WANG et al, 2022;YONEOKA et al, 2020).…”

“…Apesar do aumento no uso de chatbots em saúde, ainda há poucos estudos sobre sua aceitação, segurança e eficácia. Adicionalmente, a ausência de padrões e a escassez de métricas objetivas tornam a comparação do desempenho dos chatbots na área da saúde mais desafiadora (CHAGAS et al, 2023). Desse modo, este trabalho objetiva analisar, por meio da literatura disponível, o uso de chatbots para a triagem de sintomas da COVID-19.…”

Head Protection Device for Individuals at Risk for Head Injury due to Ground-Level Falls: Single Trauma Center User Experience Investigation

“…Human factors, a scientific discipline, is important in clinical practice as it reveals how humans interact with interventions, such as devices, regarding expectations and limitations. User experience (UX) focuses on having a deep understanding of users and what they need and value [21][22][23]. UX research has been used to ascertain user domains such as adherence, usability, and perceived impact and has assisted with intervention development and refinement [24].…”