A stochastic model of human-machine interaction for learning dialog strategies

Abstract

In this paper, we propose a quantitative model for dialog systems that can be used for learning the dialog strategy. We claim that the problem of dialog design can be formalized as an optimization problem with an objective function reflecting different dialog dimensions relevant for a given application. We also show that any dialog system can be formally described as a sequential decision process in terms of its state space, action set, and strategy. With additional assumptions about the state transition probabilities and cost assignment, a dialog system can be mapped to a stochastic model known as Markov decision process (MDP). A variety of data driven algorithms for finding the optimal strategy (i.e., the one that optimizes the criterion) is available within the MDP framework, based on reinforcement learning. For an effective use of the available training data we propose a combination of supervised and reinforcement learning: the supervised learning is used to estimate a model of the user, i.e., the MDP parameters that quantify the user's behavior. Then a reinforcement learning algorithm is used to estimate the optimal strategy while the system interacts with the simulated user. This approach is tested for learning the strategy in an air travel information system (ATIS) task. The experimental results we present in this paper show that it is indeed possible to find a simple criterion, a state space representation, and a simulated user parameterization in order to automatically learn a relatively complex dialog behavior, similar to one that was heuristically designed by several research groups.

A statistical approach to spoken dialog systems design and evaluation

“…The most extended methodology for machine-learning of dialog strategies consists of modeling human-computer interaction as an optimization problem using Markov Decision Process (MDP) and reinforcement methods , (Singh et al, 1999), (Levin et al, 2000). The main drawback of this approach is due to the large state space of practical spoken dialog systems, whose representation is intractable if represented directly .…”

“…Models of this kind have been widely used for speech recognition and also for language understanding (Levin and Pieraccini, 1995), (Minker et al, 1999), (Segarra et al, 2002), (He and Young, 2003), (Esteve et al, 2003). Even though in the literature there are models for dialog managers that are manually designed, over the last few years, approaches using statistical models to represent the behavior of the dialog manager have also been developed (Levin et al, 2000), (Torres et al, 2003), , (Williams and Young, 2007). These approaches are usually based on modeling the different processes probabilistically and learning the parameters of the different statistical models from a dialog corpus.…”

“…Its weak point consists of approximating the complete history of the dialog by a bigram model. In (Levin et al, 2000), the bigram model is modified by considering only a set of possible user answers following a given system action (the Levin model). Both models have the drawback of considering that every user response depends only on the previous system turn.…”

Language Generation for Conversational Agent by Evolution of Plan Trees with Genetic Programming

“…Walker et al proposed the sentence plan tree representing the structure of a sentence while training is conducted by a RankBoost method [4]. A stochastic model generating sentences was presented by Levin et al while they have applied it to a dialogue-based travel planning system [5]. And Ratnaparkhi has suggested a hybrid model [3], while Bulyko and Ostendorf generated various sentences using a weighted finite state machine [6].…”

“…In order to overcome the limitations, trainable approaches have been attempted recently [3,4,5,6]. Walker et al proposed the sentence plan tree representing the structure of a sentence while training is conducted by a RankBoost method [4].…”