User preferences can drive facial expressions: evaluating an embodied conversational agent in a recommender dialogue system

Abstract

Tailoring the linguistic content of automatically generated descriptions to the preferences of a target user has been well demonstrated to be an effective way to produce higher-quality output that may even have a greater impact on user behaviour. It is known that the non-verbal behaviour of an embodied agent can have a significant effect on users' responses to content presented by that agent. However, to date no-one has examined the contribution of non-verbal behaviour to the effectiveness of user tailoring in automatically generated embodied output. We describe a series of experiments designed to address this question. We begin by introducing a multimodal dialogue system designed to generate descriptions and comparisons tailored to user preferences, and demonstrate that the user-preference tailoring is detectable to an overhearer when the output is presented as synthesised speech. We then present a multimodal corpus consisting of the annotated facial expressions used by a speaker to accompany the generated tailored descriptions, and verify that the most characteristic positive and negative expressions used by that speaker are identifiable when resynthesised on an artificial talking head. Finally, we combine the corpus-derived facial displays with the tailored descriptions to test whether the addition of the non-verbal channel improves users' ability to detect the intended tailoring, comparing two strategies for selecting the displays: one based on a simple corpus-derived rule, and one This article integrates and extends the work described in Foster and White (2005) and Foster (2007a,b). M. E. Foster (B)123 342 M. E. Foster, J. Oberlander making direct use of the full corpus data. The performance of the subjects who saw displays selected by the rule-based strategy was not significantly different than that of the subjects who got only the linguistic content, while the subjects who saw the data-driven displays were significantly worse at detecting the correctly tailored output. We propose a possible explanation for this result, and also make recommendations for developers of future systems that may make use of an embodied agent to present user-tailored content.

Unleashing the Potential of Chatbots in Education: A State-Of-The-Art Analysis

“…Especially, detecting knowledge gaps of students is essential to increase cognitive learning outcomes of students (Pérez et al, 2016). Moreover, detecting the emotional status and including relational behavior such as caring and empathy helps students to improve their affective learning outcomes (Foster & Oberlander, 2010). This is also the reason why chatbots should include small talk (Bickmore et al, 2013;Kerly et al, 2007).…”

A Survey on Conversational Recommender Systems

“…Various factors can impact the effectiveness of such ECAs. In [42], for example, the authors analyze the effects of non-verbal behavior (e.g., the facial expressions) on the effectiveness of an ECA in the context of a dialogue-based recommender system. Research on the specific effects of using different variants of an ECA in the context of recommender systems is, however, generally rare.…”

“…Often, the CRS is implemented in the form of chatbot that is embedded within e-commerce solutions [32,156] or other types of web-portals [21]. In some cases, the CRS is also part of a multi-modal 2D or 3D user experience, like in [42] and [33]. A special case in this context is the use of a CRS on voice-based home assistants (smart speakers) [4,36].…”

“…Providing more information about a recommended item, e.g., in the form of pros and cons, is a typical approach when providing explanations. Generating such item descriptions in a user-tailored way in the context of CRS was proposed for example in [142] and [42]. In such approaches, the users' feedback during the conversation can influence which attributes are mentioned in the item descriptions shown to the user in the recommendation phase.…”

“…In such approaches, the users' feedback during the conversation can influence which attributes are mentioned in the item descriptions shown to the user in the recommendation phase. Furthermore, the user preferences can be considered to order the arguments and to help determining which adjectives and adverbs to use in the explanation [42].…”