Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning

Abstract

Developing a Dialogue/Virtual Agent (VA) that can handle complex tasks (need) of the user pertaining to multiple intents of a domain is challenging as it requires the agent to simultaneously deal with multiple subtasks. However, majority of these end-to-end dialogue systems incorporate only user semantics as inputs in the learning process and ignore other useful user behavior and information. Sentiment of the user at the time of conversation plays an important role in securing maximum user gratification. So, incorporating sentiment of the user during the policy learning becomes even more crucial, more so when serving composite tasks of the user. Methodology As a first step towards enabling the development of sentiment aided VA for multi-intent conversations, this paper proposes a new dataset, annotated with its corresponding intents, slot and sentiment (considering the entire dialogue history) labels, named SentiVA, collected from open-sourced dialogue datasets. In order to integrate these multiple aspects, a Hierarchical Reinforcement Learning (HRL) specifically options based VA is proposed to learn strategies for managing multi-intent conversations. Along with task success based immediate rewards, sentiment based immediate rewards are also incorporated in the hierarchical value functions to make the VA user adaptive. Findings Empirically, the paper shows that task based and sentiment based immediate rewards cumulatively are required to ensure successful task completion and attain maximum user satisfaction in a multi-intent scenario instead of any of these rewards alone. Practical implications The eventual evaluators and consumers of dialogue systems are users. Thus, to ensure a fulfilling conversational experience involving maximum user satisfaction requires VA to consider user sentiment at every time-step in its decision making policy.

A dynamic goal adapted task oriented dialogue agent

“…It might not be entirely fair to compare our VA with some traditional task oriented dialogue agents as they have not been trained for such scenarios. Still, to establish the importance and efficacy of the proposed system, we experimented with a few recent task-oriented dialogue systems [ 17 , 20 , 30 , 41 ] for the proposed problem. n [ 17 ], authors proposed a simple yet effective methodology for optimizing dialogue policy using reinforcement learning.…”

“…In [ 41 ], the authors proposed a Natural Language Understanding (NLU) robust Goal Oriented Bot (GO-Bot) for movie ticket booking. In [ 30 ], the authors presented a Sentiment aware Virtual agent (SentiVA) that establishes the importance of immediate sentiment-based reward in a multi-intent dialogue setting using Hierarchical Reinforcement Learning (HRL). These models’ performances on the proposed problem are reported in Tables 9 – 11 .…”

“…User sentiment can be treated as an explicit and grounded user feedback towards the agent’s behavior; henceforth, it can be utilized in the dialogue policy learning process to assist end-users in a more appropriate and personalized manner. In [ 30 ], authors have proposed a Hierarchical Reinforcement Learning (HRL) based agent for multi-intent dialogue setting. They have shown that sentiment reward, in addition to task-based reward, leads to a more efficient dialogue policy that ensures successful task completion with maximum user satisfaction.…”

Gaussian Process based Deep Dyna-Q approach for Dialogue Policy Learning

“…There are four typical methods, including rule based method (Weizenbaum, 1966), retrieval based method (Mikolov et al, 2013;Pennington et al, 2014;Serban et al, 2017), supervised learning based method (Sukhbaatar et al, 2015;Weston et al, 2016), and reinforcement learning based method (Levin et al, 2002). Since the reinforcement learning based method can fine-tune the current dialogue strategy based on users' feedback to promote user satisfaction, it has been the mainstream of dialogue policy learning method in recent years (Chen et al, 2020;Saha et al, 2020;Li et al, 2020).…”

“…Reinforcement learning (RL) becomes the mainstream dialogue policy learning method in recent years (Chen et al, 2020;Saha et al, 2020;Li et al, 2020). Based on the RL, the task-completion dialogue system can gradually adjust policy through interacting with real users to improve performance.…”