Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations

Abstract

Dialogue level quality estimation is vital for optimizing data driven dialogue management. Current automated methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce the generalizability of the trained models. We propose a novel user satisfaction estimation approach which minimizes an adaptive multi-task loss function in order to jointly predict turn-level Response Quality labels provided by experts and explicit dialogue-level ratings provided by end users. The proposed BiLSTM based deep neural net model automatically weighs each turn's contribution towards the estimated dialogue-level rating, implicitly encodes temporal dependencies, and removes the need to hand-craft features.On dialogues sampled from 28 Alexa domains, two dialogue systems and three user groups, the joint dialogue-level satisfaction estimation model achieved up to an absolute 27% (0.43 → 0.70) and 7% (0.63 → 0.70) improvement in linear correlation performance over baseline deep neural net and benchmark Gradient boosting regression models, respectively.

A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction Analysis

“…Multi-task learning in dialogue system. For satisfaction estimation, Bodigutla et al (2020) propose to jointly predict turn-level RQ labels and dialogue-level ratings. They utilize features from spoken dialogue system and BiLSTM (Hochreiter and Schmidhuber, 1997) based model to automatically weight each turn's contribution towards the rating.…”

“…This example illustrates the cross-impact between handoff and dialogue (local+global) satisfaction. Intuitively, MHCH and SSA tasks can be compatible and complementary given a dialogue discourse, i.e., the local satisfaction is related to the quality of the conversation (Bodigutla et al, 2019a(Bodigutla et al, , 2020, which can support the handoff judgment and ultimately affect the overall satisfaction. On the one hand, handoff labels of utterances are highly pertinent to local satisfaction, e.g., one can utilize single handoff information to enhance local satisfaction prediction, which ultimately contributes to the overall satisfaction estimation.…”

“…In recent years, researchers (Bodigutla et al, 2019a,b;Ultes, 2019;Bodigutla et al, 2020) explore joint evaluation of turn and dialogue level qualities in spoken dialogue systems. In terms of general dialogue system, to improve the efficiency of dialogue management, Qin et al (2020) propose a co-interactive relation layer to explicitly examine the cross-impact and model the interaction between sentiment classification and dialog act recognition, which are relevant tasks at the same level (utterancelevel).…”

Turn-Level User Satisfaction Estimation in E-commerce Customer Service

“…While some works also addressed the turn-level online satisfaction estimation, they needed turnlevel human annotations (Ultes et al, 2017;Bodigutla et al, 2020). These methods are not scalable in terms of annotation costs due to the large volumes of user data in E-commerce.…”

User Satisfaction Reward Estimation Across Domains: Domain-independent Dialogue Policy Learning

“…A similar measure called response quality has been proposed as a measure to capture user satisfaction as an alternative to interaction quality (Bodigutla et al, 2019b(Bodigutla et al, ,a, 2020. In contrast to the interaction quality, the response quality focuses more on the overal performance of a system, e.g., including the functionality of back-end services.…”