A two-tier user simulation model for reinforcement learning of adaptive referring expression generation policies

Abstract

We present a new two-tier user simulation model for learning adaptive referring expression generation (REG) policies for spoken dialogue systems using reinforcement learning. Current user simulation models that are used for dialogue policy learning do not simulate users with different levels of domain expertise and are not responsive to referring expressions used by the system. The twotier model displays these features, that are crucial to learning an adaptive REG policy. We also show that the two-tier model simulates real user behaviour more closely than other baseline models, using the dialogue similarity measure based on Kullback-Leibler divergence.

Towards Integration of Cognitive Models in Dialogue Management: Designing the Virtual Negotiation Coach Application

“…While early studies (Walker et al, 1998;Singh et al, 2002) used RL to build strategies for simple systems, more complex paradigms are represented by statistical models, see Frampton and Lemon (2009). However, when users with different personalities in different states are systematically confronted with a learning system, most studies resort to user simulation: Janarthanam and Lemon (2009) These studies demonstrate that the simulation of different user types is expected to lead to strategies which adapt to each user type. However, adaptivity has been not achieved at the level of dynamically changing goals within one dialogue.…”

A survey on metrics for the evaluation of user simulations

“…User simulation is required to expand data sets used for training RL-based dialogue managers Singh et al, 1999;Scheffler & Young, 2001;Pietquin, 2004;Williams et al, 2005) and natural language generation systems (Janarthanam & Lemon, 2009a, 2009b, 2009c. User simulation is required to expand data sets used for training RL-based dialogue managers Singh et al, 1999;Scheffler & Young, 2001;Pietquin, 2004;Williams et al, 2005) and natural language generation systems (Janarthanam & Lemon, 2009a, 2009b, 2009c.…”

“…In order to do so, it is necessary to provide a clear idea of the purpose of a user simulation. User simulation is required to expand data sets used for training RL-based dialogue managers (Levin et al ., 1997; Singh et al ., 1999; Scheffler & Young, 2001; Pietquin, 2004; Williams et al ., 2005) and natural language generation systems (Janarthanam & Lemon, 2009a, 2009b, 2009c). This provides at least two requirements for the user simulation evaluation metric: it should assess how well the simulation fits the original data statistics ( consistency ) and it should result in efficient strategies when used for training RL-based systems ( quality of learnt strategy ).…”

“…From the mid 90s, user simulation has become an important area of research in the field of spoken dialogue systems (SDSs) because collecting and annotating real human–machine interactions is often expensive and time-consuming (Eckert et al ., 1997; Zukerman & Albrecht, 2001; Cuayahuitl et al ., 2005; Georgila et al ., 2005; Pietquin, 2006; Schatzmann et al ., 2007b; Janarthanam & Lemon, 2009b; Pietquin et al ., 2009). Such data are generally required for designing, training and assessing dialogue systems (Levin et al ., 2000; Scheffler & Young, 2001; López-Cózar et al ., 2003; Pietquin & Dutoit, 2006; Schatzmann et al ., 2007a).…”

Assessing user simulation for dialog systems using human judges and automatic evaluation measures

“…A table of critical values is also given to interpret the statistical significance of comparisons between different corpora. A similar approach is used in Janarthanam and Lemon (2009) and Jung et al (2009) as well. They argue that the goal of evaluating the quality of a simulated corpus is not to measure how well the simulation models can resemble the behavior of an average user.…”