User Experience of a Chatbot Questionnaire Versus a Regular Computer Questionnaire: Prospective Comparative Study

Abstract

Background Respondent engagement of questionnaires in health care is fundamental to ensure adequate response rates for the evaluation of services and quality of care. Conventional survey designs are often perceived as dull and unengaging, resulting in negative respondent behavior. It is necessary to make completing a questionnaire attractive and motivating. Objective The aim of this study is to compare the user experience of a chatbot questionnaire, which mimics intelligent conversation, with a regular computer questionnaire. Methods The research took place at the preoperative outpatient clinic. Patients completed both the standard computer questionnaire and the new chatbot questionnaire. Afterward, patients gave their feedback on both questionnaires by the User Experience Questionnaire, which consists of 26 terms to score. Results The mean age of the 40 included patients (25 [63%] women) was 49 (SD 18-79) years; 46.73% (486/1040) of all terms were scored positive for the chatbot. Patients preferred the computer for 7.98% (83/1040) of the terms and for 47.88% (498/1040) of the terms there were no differences. Completion (mean time) of the computer questionnaire took 9.00 minutes by men (SD 2.72) and 7.72 minutes by women (SD 2.60; P=.148). For the chatbot, completion by men took 8.33 minutes (SD 2.99) and by women 7.36 minutes (SD 2.61; P=.287). Conclusions Patients preferred the chatbot questionnaire over the computer questionnaire. Time to completion of both questionnaires did not differ, though the chatbot questionnaire on a tablet felt more rapid compared to the computer questionnaire. This is an important finding because it could lead to higher response rates and to qualitatively better responses in future questionnaires.

How the Wakamola chatbot studied a university community’s lifestyle during the COVID-19 confinement

“…Wambsganss et al 54 found that participants using chatbot surveys showed higher response quality and social presence than a web survey. In a health context, Te Pas et al 55 concluded that patients preferred a chatbot questionnaire over a computer questionnaire as the chatbot was faster, although real-time to completion of both questionnaires did not differ. Based on these studies, chatbots are a feasible alternative to traditional online and computer surveys to improve collected data quality and user experience.…”

“…Future Wakamola chatbot improvement should include questions about the quality of sleep, for example, to allow calculation of the Pittsburgh Sleep Quality Index (PSQI), 55 for a better understanding of sleep changes during confinement. 65…”

Towards Chatbot-Supported Self-Reporting for Increased Reliability and Richness of Ground Truth for Automatic Pain Recognition: Reflections on Long-Distance Runners and People with Chronic Pain

“…This is in line with previous research [30] which argued that a sense of a socially shared experience contributes to a higher compliance rate of ESM tasks. As pointed in existing literature, traditional computergenerated questionnaire is dull and lacking elements to motivate participants, whereas a chatbot-delivered questionnaire could provide more engaging conversational interactions, which can lead to higher quantity and quality of participants' responses to the ESM questions [31].…”

Assessing the Feasibility of Studying Awareness of a Digital Health Campaign on Facebook: Pilot Study Comparing Young Adult Subsamples

“…Recruitment of participants and exposure to target advertisements occur on Facebook. The surveys are disseminated within Facebook Messenger using a chatbot questionnaire, which has been shown to increase user engagement, user satisfaction, and data quality over traditional web surveys [ 17 - 19 ]. The feasibility of Virtual Lab would allow for quick recruitment of participants and delivery of results, as well as easy navigation of a dashboard during data collection, while remaining cost-effective.…”