Nonstrict Hierarchical Reinforcement Learning for Interactive Systems and Robots

Abstract

Conversational systems and robots that use reinforcement learning for policy optimization in large domains often face the problem of limited scalability. This problem has been addressed either by using function approximation techniques that estimate the approximate true value function of a policy or by using a hierarchical decomposition of a learning task into subtasks. We present a novel approach for dialogue policy optimization that combines the benefits of both hierarchical control and function approximation and that allows flexible transitions between dialogue subtasks to give human users more control over the dialogue. To this end, each reinforcement learning agent in the hierarchy is extended with a subtask transition function and a dynamic state space to allow flexible switching between subdialogues. In addition, the subtask policies are represented with linear function approximation in order to generalize the decision making to situations unseen in training. Our proposed approach is evaluated in an interactive conversational robot that learns to play quiz games. Experimental results, using simulation and real users, provide evidence that our proposed approach can lead to more flexible (natural) interactions than strict hierarchical control and that it is preferred by human users.

Scaling up deep reinforcement learning for multi-domain dialogue systems

“…Although the currently generated dialogues using the trained policies seem reasonable 6 , it is natural to ask How good are the DRL-based policies? To answer this question we integrated a K-Nearest Neighbour (KNN) baseline [13], which aims to behave as the example demonstration dialogues-see Appendix A.…”

“…In contrast to Hierarchical DQNs [5] that follow a strict sequence of agents, an NDQN in our method allows transitions between all DQN agents except for self-transitions. The latter using a stack-based approach as in [6]. While user responses can motivate transitions to another domain in the network, completing a subdialogue within a domain motivates a transition to the previous domain to resume the interaction.…”

SimpleDS: A Simple Deep Reinforcement Learning Dialogue System

“…It remains to be demonstrated how far one can go with such an approach. Future work includes to (a) compare different model architectures, training parameters and reward functions; (b) extend or improve the abilities of the proposed dialogue system; (c) train deep learning agents in other (larger scale) domains [7,8,9]; (d) evaluate end-to-end systems with real users; (e) compare or combine different types of neural nets [10]; and (e) perform fast learning based on parallel computing. Table 1 Example dialogue using the policy from Fig.2, where states are numerical representations of the last system and noisy user inputs, actions are dialogue acts, and user resposes are in brackets…”

Introduction to the Special Issue on Machine Learning for Multiple Modalities in Interactive Systems and Robots

“…This function is known as a classifier when the labels are discrete and as a regressor when the labels are continuous. All articles in this special issue make use of classifiers to predict events during human-machine interactions [Ngo et al 2014;Benotti et al 2014;Keizer et al 2014;Cuayáhuitl et al 2014]. -In contrast to supervised learning that makes use of direct feedback, reinforcement learning makes use of indirect feedback typically based on numerical rewards given during the interaction, and the goal is to maximize them in the long run.…”