Detecting Egregious Conversations between Customers and Virtual Agents

Abstract

Virtual agents are becoming a prominent channel of interaction in customer service. Not all customer interactions are smooth, however, and some can become almost comically bad. In such instances, a human agent might need to step in and salvage the conversation. Detecting bad conversations is important since disappointing customer service may threaten customer loyalty and impact revenue. In this paper, we outline an approach to detecting such egregious conversations, using behavioral cues from the user, patterns in agent responses, and useragent interaction. Using logs of two commercial systems, we show that using these features improves the detection F1-score by around 20% over using textual features alone. In addition, we show that those features are common across two quite different domains and, arguably, universal.

Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems

“…Based on these limitations, our proposed solution is to apply data programming to generate training data by using heuristic weak supervision strategies. We combine our domain heuristics to design a set of simple rule-based labeling functions [27,28] to generate online training labels. Once large-scale training data is generated, the goal is to compare heuristic performance with proposed models to see if models can learn beyond these simple rules.…”

“…One recent work proposed a query representation learning technique with intent-sensitive word embeddings, and showed that modifications to improve query representation can improve overall model performance [16]. Another recent work introduced a model that can detect egregious conversations using textual representations, and addressed how this technique can be applied to an automated evaluation scheme [28]. There have been studies to predict causes of query reformulation in intelligent assistants by using system, acoustic, language and additional features [29].…”

“…For the experiments in this paper, we chose to frame the problem as a binary classification task, to predict SAT (satisfaction) vs. DSAT (dis-satisfaction). There is a long tradition in evaluation literature for this approach, e.g., [15,16,20,21,28] in order to reduce high subjectivity and noise in user ratings. The challenge is where to choose the boundary to convert the user ratings to SAT/DSAT decisions.…”

“…Evaluating intelligent assistants is a challenging task, and has been an active area of research. For example, recent studies identified particular patterns of interactions which tend to contribute to final user satisfaction (e.g., [16,21,28]), and new behavioral metrics such as conversational depth and topic diversity have been proposed to systematically evaluate user experience in conversational systems [20,26,32]. However, as we will show, these metrics do not directly correspond to actual subjective and immediate user satisfaction with the conversation.…”

Quantifying the Effects of Prosody Modulation on User Engagement and Satisfaction in Conversational Systems

“…As these systems became more sophisticated, many work proposed new ideas to automate the evaluation process by predicting conversational user satisfaction, as defined in [28][29][30]. For instance, there have been successful attempts to predict satisfaction once conversations (sessions) are completed, using traditional methods [17,22] and neural-based models [13,14]. Lastly, one recent work [5] proposed a unified neural framework to predict offline (sessionlevel) and online (turn-level) satisfaction simultaneously.…”