A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies

Abstract

Within the broad field of spoken dialogue systems, the application of machine-learning approaches to dialogue management strategy design is a rapidly growing research area. The main motivation is the hope of building systems that learn through trial-and-error interaction what constitutes a good dialogue strategy. Training of such systems could in theory be done using human users or using corpora of human–computer dialogue, but in practice the typically vast space of possible dialogue states and strategies cannot be explored without the use of automatic user simulation tools.This requirement for training statistical dialogue models has created an interesting new application area for predictive statistical user modelling and a variety of different techniques for simulating user behaviour have been presented in the literature ranging from simple Markov models to Bayesian networks. The development of reliable user simulation tools is critical to further progress on automatic dialogue management design but it holds many challenges, some of which have been encountered in other areas of current research on statistical user modelling, such as the problem of ‘concept drift’, the problem of combining content-based and collaboration-based modelling techniques, and user model evaluation. The latter topic is of particular interest, because simulation-based learning is currently one of the few applications of statistical user modelling that employs both direct ‘accuracy-based’ and indirect ‘utility-based’ evaluation techniques.In this paper, we briefly summarize the role of the dialogue manager in a spoken dialogue system, give a short introduction to reinforcement-learning of dialogue management strategies and review the literature on user modelling for simulation-based strategy learning. We further describe recent work on user model evaluation and discuss some of the current research issues in simulation-based learning from a user modelling perspective.

A statistical approach to spoken dialog systems design and evaluation

“…In the literature, there are several corpus-based approaches for developing user simulators, learning optimal management strategies, and evaluating the dialog system (Scheffler and Young, 2001a) (Pietquin and Dutoit, 2005) (Cuayáhuitl et al, 2006). A summary of user simulation techniques for reinforcement learning of the dialog strategy can be found in (Schatzmann et al, 2006).…”

“…Typically used methods are adopted from other research fields such as Information Retrieval and Machine Learning. A first classification consists of dividing these techniques into direct evaluation methods and indirect methods (Schatzmann et al, 2006).…”

“…In (Scheffler and Young, 2001b) and (Schatzmann et al, 2006), a set of statistical measures to evaluate the quality of the simulated corpus is proposed. Three dimensions are defined: high-level features (dialog and turn lengths), dialog style (speech-act frequency; proportion of goal-directed actions, grounding, formalities, and unrecognized actions; proportion of information provided, reprovided, requested and rerequested), and dialog efficiency (goal completion rates and times).…”

Agent Simulation to Develop Interactive and User-Centered Conversational Agents

“…This way, the development of multi-agent systems offers the capability of simulating autonomous agents and the interaction between them. In the literature, there are several corpus-based approaches for developing user simulators, learning optimal dialog strategies, and evaluating conversational agents [10,9,6,3]. The construction of user models based on statistical methods has provided interesting and well-founded results in recent years and is currently a growing research area.…”

“…Therefore, it can be used to learn a dialog strategy by means of its interaction with the conversational agent. A summary of user simulation techniques for reinforcement learning of the dialog strategy can be found in [9].…”