Coerced Change-talk with Conversational Agents Promotes Confidence in Behavior Change

Ólafsson, Stefán; O’Leary, Teresa K.; Bickmore, Timothy

doi:10.1145/3329189.3329202

Cited by 30 publications

(76 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

Section: Metrics Related To Response Generationmentioning

confidence: 99%

“…In total, 14 (21.5%) studies assessed how human-like chatbots are (realism) [17,18,21,34,39,41,46,50,63,66,68,[70][71][72]. Realism of chatbots was examined in terms of verbal responses only [17,21,34,39,46,63,68,70], nonverbal responses only [66], or both verbal and nonverbal responses [18,41,50,71,72]. The included studies evaluated realism using a single question in a self-administrated questionnaire [17,18,21,39,46,50,63,70], multiple questions in a self-administrated questionnaire [41,72], and interviews [18,34,66,68,71].…”

Section: Metrics Related To Response Generationmentioning

confidence: 99%

“…The speed of responses was examined using a single question in a self-administrated questionnaire [18,30,36,69,70,73], interviews [19,34,68], multiple questions in a self-administrated questionnaire [53], and system logs [38]. Empathy of a chatbot, which refers to its ability to show empathy to users, was examined in 10 studies [7,35,41,42,64,66,67,71,73,74]. Those studies evaluated empathy using a single question in a self-administrated questionnaire [7,35,41,42,67,71,73], interviews [66,71], and multiple questions in a self-administrated questionnaire [64].…”

Section: Metrics Related To Response Generationmentioning

confidence: 99%

See 2 more Smart Citations

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

et al. 2020

View full text Add to dashboard Cite

Background Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field. Objective This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots. Methods Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated. Results Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content). Conclusions The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

show abstract

Section: Metrics Related To Response Generationmentioning

confidence: 99%

Section: Metrics Related To Response Generationmentioning

confidence: 99%

Section: Metrics Related To Response Generationmentioning

confidence: 99%

See 1 more Smart Citation

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Computeranimated agents are responsive to this problem in that they deliver information on personally relevant screening topics and interact with women to provide emotional support. They have been shown to reduce anxiety, boost confidence, 31 and support shared decision-making 32 across health topics. Most mammography results dissemination continues to be done outside of the context of a visit, does not satisfy women's informational needs, 44,45 and is accompanied by provider reticence to comprehensively counsel on breast density, [16][17][18] including risk assessment.…”

Section: Discussionmentioning

confidence: 99%

“…Based on prior work developing conversational agent systems that are responsive to patient health information needs, [29][30][31][32] we created a prototype computer-animated agent that simulates a face-to-face conversation with a health educator (Fig. 1).…”

Section: Overviewmentioning

confidence: 99%