Hierarchical reinforcement learning for situated natural language generation

Abstract

Natural Language Generation systems in interactive settings often face a multitude of choices, given that the communicative effect of each utterance they generate depends crucially on the interplay between its physical circumstances, addressee and interaction history. This is particularly true in interactive and situated settings. In this paper we present a novel approach forsituated Natural Language Generationin dialogue that is based onhierarchical reinforcement learningand learns the best utterance for a context by optimisation through trial and error. The model is trained from human–human corpus data and learns particularly to balance the trade-off betweenefficiencyanddetailin giving instructions: the user needs to be given sufficient information to execute their task, but without exceeding their cognitive load. We present results from simulation and a task-based human evaluation study comparing two different versions of hierarchical reinforcement learning: One operates using a hierarchy of policies with a large state space and local knowledge, and the other additionally shares knowledge across generation subtasks to enhance performance. Results show that sharing knowledge across subtasks achieves better performance than learning in isolation, leading to smoother and more successful interactions that are better perceived by human users.

Scaling up deep reinforcement learning for multi-domain dialogue systems

“…To our knowledge, we report one of the first multi-domain dialogue system using deep reinforcement learning. Future work includes applying neural-based dialogue systems to larger sets of domains, to language generation using a divide-and-conquer approach [28], to multi-task multimodal interaction using different types of devices and machines, and to evaluate neural-based systems in realistic scenarios with genuine users.…”

Nonstrict Hierarchical Reinforcement Learning for Interactive Systems and Robots

“…Similarly, gestures can be jointly optimized with natural language generation to exhibit more natural multimodal output. For other kinds of joint optimizations, see [Dethlefs et al 2012b;Dethlefs and Cuayáhuitl 2011b;2011a;Lemon 2011]. (5) Optimize dialogue control by adding linguistically-motivated phenomena such as multimodal turn-taking [Chao and Thomaz 2012; and incremental processing [Schlangen and Skantze 2009;Dethlefs et al 2012a;.…”

“…Especially, the decomposition of behavior into subbehaviors leads to faster learning, reduces computational demands, provides opportunities for the reuse of subsolutions, and is more suitable for multi-task systems. The utility of hierarchical RL typically increases with the complexity and size of the state-action space of a system and has also been adopted for dialogue in combination with robust speech recognition [Pineau 2004] and natural language generation [Dethlefs and Cuayáhuitl 2010;. However, previous works in this area have ignored interaction flexibility.…”

Context Analysis of Customer Requests using a Hybrid Adaptive Neuro Fuzzy Inference System and Hidden Markov Models in the Natural Language Call Routing Problem

“…Dethlefs N. and Cuayáhuitl, H. presented novel approach for situated Natural Language Generation in dialogue that is based on hierarchical reinforcement learning and learns the best utterance for a context by optimisation through trial and error [26]. Ferreira, T. C et al introduced nondeterministic method for referring expression generation.…”