A Methodology for Evaluating Interaction Strategies of Task-Oriented Conversational Agents

Abstract

In task-oriented conversational agents, more attention has been usually devoted to assessing task effectiveness, rather than to how the task is achieved. However, conversational agents are moving towards more complex and human-like interaction capabilities (e.g. the ability to use a formal/informal register, to show an empathetic behavior), for which standard evaluation methodologies may not suffice. In this paper, we provide a novel methodology to assess-in a completely controlled way-the impact on the quality of experience of agent's interaction strategies. The methodology is based on a within subject design, where two slightly different transcripts of the same interaction with a conversational agent are presented to the user. Through a series of pilot experiments we prove that this methodology allows fast and cheap experimentation/evaluation, focusing on aspects that are overlooked by current methods.

Toward Stance-based Personas for Opinionated Dialogues

“…formal vs. informal register) as used in goal-oriented settings by Joshi et al (2017); Luo et al (2019) to provide personalized interactions. Further, Guerini et al (2018) showed how injecting these specific persona-related aspects into a conversation can positively affect the interaction in goal-oriented scenarios, both in terms of quality of service and overall perceived quality.…”