Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

Abstract

Background Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. Objective This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. Methods Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. Results Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). Conclusions The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

The evaluation of chatbot as a tool for health literacy education among undergraduate students

“…No-match information in Dialogflow is when one of the fallback intents is triggered. This is also aligned with the study by Abd-Alrazaq et al (2020), where they concluded that a chatbot should be evaluated based on usability, performance, responses, and esthetics.…”

“…The technical design, linguistic challenges, and emotional intelligence conversation should be designed carefully for a chatbot. According to Abd-Alrazaq et al (2020), the chatbot should be evaluated based on usability, performance, responses, and esthetics. However, the experts expressed their caution of using the chatbot technology in health-related activities, as stated by John et al (2020) in their study.…”

Artificial Intelligence Chatbot Behavior Change Model for Designing Artificial Intelligence Chatbots to Promote Physical Activity and a Healthy Diet: Viewpoint

“…JITAI designs in combination with real-time data from ecological momentary assessment, in-built accelerometers, GPS, and/or other sensors will allow chatbots to customize the timing, amount, content, and frequency of the intervention, by adapting each individual’s internal and external changes over time. However, a recent scoping review of health care chatbots showed that the use of JITAIs in designing and evaluating chatbots in health care in general and promoting physical activity and a healthy diet in particular is sparse, suggesting that future research needs to consider using more of these adaptive approaches [ 89 ].…”

Perceptions and Opinions of Patients About Mental Health Chatbots: Scoping Review

“…Evaluation measures were divided into 3 main types: technical performance, user experience, and health research measures. The first attempts toward evaluation frameworks for digital health interventions [ 82 ] and health chatbots [ 83 , 84 ] have been recently published. Depending on the facet under consideration, different metrics can be used: system performance and effectiveness can be evaluated using different computational metrics (eg, usability, ease of use, usefulness) [ 85 ].…”