Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design

Haugeland, Isabel Kathleen Fornell; Følstad, Asbjørn; Taylor, Cameron; Bjørkli, Cato Alexander

doi:10.1016/j.ijhcs.2022.102788

Cited by 75 publications

(33 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The chatbots are typically set up utilising what [38] refers to as statistical data-driven approaches where user intents are inferred on the basis of underlying AI-based prediction models. Users are typically encouraged to enter their requests in free text from which the specific user intent is predicted [25]. Based on the predicted user intent, the chatbot provides the corresponding action, typically sending the user one or more messages conveying relevant content to their request.…”

Section: Background 21 Chatbots For Customer Servicementioning

confidence: 99%

“…trust). Chatbot design features found to impact humanlikeness and perceptions of anthropomorphism include conversational style [25], visual representation and initial self-presentation [2,20], informal language [2], and features hinting at chatbot intelligence such as backchanneling [20] and conversational relevance [50]. In turn, chatbot humanlikeness influences user perception and behaviour on a range of dimensions, including hedonic user experience [25], brand perception [2], user sentiment [10], user compliance [1], transaction conversion [48], and intention to use [33].…”

Section: Chatbot Humanlikenessmentioning

confidence: 99%

“…For example, humanlike elements like pronouns, greetings or even jokes and emoji may be necessary to make sure that the conversations are consistent or that the replies fit in with the general personality that the chatbot is supposed to have [13]. Alternatively, if the goal is to improve hedonic experiences when people chat with the bot, adding humanlikeness may also be a good way to spend one's time [25]. However, it is crucial that teams working with chatbots understand that it will not necessarily lead to higher trust in end-users, nor that it will help their bot become more trust resilient in the face of conversational breakdown.…”

Section: Practical Implicationsmentioning

confidence: 99%

See 2 more Smart Citations

Effects of Humanlikeness and Conversational Breakdown on Trust in Chatbots for Customer Service

Law

Følstad

As³

2022

Nordic Human-Computer Interaction Conference

Self Cite

View full text Add to dashboard Cite

Trust in chatbots can be shaped by various factors such as humanlikeness in terms of visual appearance and conversational content, and conversational performance in terms of the chatbot's ability to avoid conversational breakdown. The literature is inconclusive concerning the effect of humanlikeness and conversational performance on trust, especially their interaction effect. To examine the relations among these variables, we conducted a 2x3 (humanlikeness x conversational performance) factorial experiment with 251 participants, who were asked to perform three tasks with a chatbot for an online bank under one of the six conditions. Participants completed a questionnaire measuring trust and commented on trust factors. Results of between-group analysis showed that for the task with seeded breakdowns there were significant differences in trust across the six groups with the lowest ratings for the two groups experiencing breakdowns without repairs and that humanlikeness did not impact the extent to which the trust level changed. Results of within-group analysis showed significant differences in trust across the tasks but non-significant inter-task correlations on trust for the two groups. These observations challenge the effect of humanlikeness on trust while supporting the notion of trust resilience as the participants did not spill the impaired trust over the subsequent task. Thematic analysis showed that inter-group contrasts could be found for the theme 'underlying functionality' and 'affective responses.' Implications for research, practice and future work were drawn. CCS CONCEPTS• Human-centered computing → Empirical studies in HCI.

show abstract

Section: Background 21 Chatbots For Customer Servicementioning

confidence: 99%

Section: Chatbot Humanlikenessmentioning

confidence: 99%

Section: Practical Implicationsmentioning

confidence: 99%

See 1 more Smart Citation

Effects of Humanlikeness and Conversational Breakdown on Trust in Chatbots for Customer Service

Law

Følstad

As³

2022

Nordic Human-Computer Interaction Conference

Self Cite

View full text Add to dashboard Cite

show abstract

“…As reported by Borsci et al [5] when reviewing the domain of chatbots, it is the case that little is known about how to evaluate the end-user's perception of quality when interacting with chatbots. There is a growing interest in understanding how to assess and improve the interaction with such systems [16,22,31]; however, to our knowledge, there are currently no standardised tools to assess then enduser's satisfaction with chatbots, except for the recently developed ChatBot Usability Scale (BUS-15) [5]. The BUS-15 scale was developed and tested using an exploratory factorial analysis.…”

Section: Introductionmentioning

confidence: 99%

A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation

Borsci

Schmettow

Malizia

et al. 2022

Pers Ubiquit Comput

View full text Add to dashboard Cite

The Bot Usability Scale (BUS) is a standardised tool to assess and compare the satisfaction of users after interacting with chatbots to support the development of usable conversational systems. The English version of the 15-item BUS scale (BUS-15) was the result of an exploratory factorial analysis; a confirmatory factorial analysis tests the replicability of the initial model and further explores the properties of the scale aiming to optimise this tool seeking for the stability of the original model, the potential reduction of items, and testing multiple language versions of the scale. BUS-15 and the usability metrics for user experience (UMUX-LITE), used here for convergent validity purposes, were translated from English to Spanish, German, and Dutch. A total of 1292 questionnaires were completed in multiple languages; these were collected from 209 participants interacting with an overall pool of 26 chatbots. BUS-15 was acceptably reliable; however, a shorter and more reliable solution with 11 items (BUS-11) emerged from the data. The satisfaction ratings obtained with the translated version of BUS-11 were not significantly different from the original version in English, suggesting that the BUS-11 could be used in multiple languages. The results also suggested that the age of participants seems to affect the evaluation when using the scale, with older participants significantly rating the chatbots as less satisfactory, when compared to younger participants. In line with the expectations, based on reliability, BUS-11 positively correlates with UMUX-LITE scale. The new version of the scale (BUS-11) aims to facilitate the evaluation with chatbots, and its diffusion could help practitioners to compare the performances and benchmark chatbots during the product assessment stage. This tool could be a way to harmonise and enable comparability in the field of human and conversational agent interaction.

show abstract

“…Given the increasing use of technologies in our daily lives, user well-being -to which BPN Satisfaction could significantly contribute in the long-term -should be a key priority of developers [Peters and Calvo, 2021, Peters, 2022, Moradbakhti et al, 2022. In the context of experiments and analyses based on user-centered design, more studies could therefore focus on measures of BPN Satisfaction, specifically, if we acknowledge that technologies in many cases developed to i) become more autonomous (e.g., autonomous vehicles; Chen and Chen [2021]), ii) be more efficient and competent (e.g., algorithmic decision systems; Hou and Jung [2021]), iii) replace human contact (e.g., customer service chatbots; Haugeland et al [2022]) and iv) provide new ways of social interaction with others (e.g., social media; Roberts and David [2020]).…”

Section: Introductionmentioning

confidence: 99%

Development and Validation of a Basic Psychological Needs Scale for Technology Use

Moradbakhti¹,

Leichtmann²,

Mara³

2022

Preprint

View full text Add to dashboard Cite

The aim of this work was to develop a valid and reliable scale to measure Basic Psychological Need Satisfaction for technology use (BPN-TU). According to the Self-Determination Theory, satisfaction of the Basic Psychological Needs (BPNs) for Autonomy, Competence, and Relatedness is crucial to well-being and autonomous motivation. Research into the role of BPN Satisfaction in technology use is scarce, partly due to a lack of appropriate measuring tools. To develop a pool of original BPN-TU scale items, we held 10 interviews. Based on these items, we conducted four validation studies with four independent samples (total N = 821), collecting user responses to different technologies: digital voice assistant, exoskeleton, chatbot and social robot. Good model fit was supported by confirmatory factor analyses for a twelve-item scale, containing three items each for satisfaction of users' Autonomy, Competence, Relatedness to Others, and Relatedness to Technology. The scale was validated in English and German.

show abstract

Understanding the user experience of customer service chatbots: An experimental study of chatbot interaction design

Cited by 75 publications

References 22 publications

Effects of Humanlikeness and Conversational Breakdown on Trust in Chatbots for Customer Service

Effects of Humanlikeness and Conversational Breakdown on Trust in Chatbots for Customer Service

A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation

Development and Validation of a Basic Psychological Needs Scale for Technology Use

Contact Info

Product

Resources

About