Bayesian reinforcement learning for POMDP-based dialogue systems

Abstract

Spoken dialogue systems are gaining popularity with improvements in speech recognition technologies. Dialogue systems can be modeled effectively using POMDPs, achieving improvements in robustness. However, past research on POMDPs-based dialogue system assumes that the model parameters are known. This limitation can be addressed through model-based Bayesian reinforcement learning, which offers a rich framework for simultaneous learning and planning. However, due to the high complexity of the framework, a major challenge is to scale up these algorithms for complex dialogue systems. In this work, we show that by exploiting certain known components of the system, such as knowledge of symmetrical properties, and using an approximate online planning algorithm, we are able to apply Bayesian RL on a realistic spoken dialogue system domain.

A Comprehensive Reinforcement Learning Framework for Dialogue Management Optimization

“…However, this would cause a problem of partial observability. This could be handled by the POMDP (Partially Observed MDP) framework, but this is much more complex [25]. Instead of using this approach, we consider here the Information State Paradigm [26].…”

Convex Optimization: Algorithms and Complexity

“…The challenge in this case is to find good bounds; this is especially difficult given the uncertainty over the underlying model. The method has been used in the context of partially observable BAMDP [Png andPineau, 2011, Png, 2011] using a naive heuristic, D d=0 γ d R max , where D is the search depth and R max is the maximum reward. The method was applied successfully to solve simulated dialogue management problems; computational scalability was achieved via a number of structural constraints, including the parameter tying method proposed by .…”

“…The planning approach suggested by Ross et al [2011] aims to approximate the optimal BAPOMDP strategy by employing a forward search similar to that outlined in Algorithm 2. In related work, Png and Pineau [2011] use a branch-and-bound algorithm to approximate the BAPOMDP solution. Many of the other techniques outlined in Table 4.1 could also be extended to the BAPOMDP model.…”

“…The simplest case, called parameter tying in previous sections, corresponds to the case where states are grouped according to a pre-defined clustering assignment. In this case, it is common to aggregate learned parameters according to this assignment , Sorg et al, 2010, Png and Pineau, 2011. The advantage is that there are fewer parameters to estimate, and thus, learning can be achieved with fewer samples.…”

MACA: A Modular Architecture for Conversational Agents

“…However, the lack of tools for easy prototyping of newer models remains an impediment to developing new models and properly benchmarking against previous models. Furthermore, the different types of conversational agents-e.g., generative (Hochreiter and Schmidhuber, 1997;Serban et al, 2015Serban et al, , 2016, retrieval-based (Schatzmann et al, 2005a;Lowe et al, 2015a), slot-based (Young, 2006) or POMDP agents (Png and Pineau, 2011)have different working mechanisms, which pose challenges to the development of a unified platform for conversational agents with multi-domain support.…”