The Second Dialog State Tracking Challenge

Abstract

A spoken dialog system, while communicating with a user, must keep track of what the user wants from the system at each step. This process, termed dialog state tracking, is essential for a successful dialog system as it directly informs the system's actions. The first Dialog State Tracking Challenge allowed for evaluation of different dialog state tracking techniques, providing common testbeds and evaluation suites. This paper presents a second challenge, which continues this tradition and introduces some additional features -a new domain, changing user goals and a richer dialog state. The challenge received 31 entries from 9 research groups. The results suggest that while large improvements on a competitive baseline are possible, trackers are still prone to degradation in mismatched conditions. An investigation into ensemble learning demonstrates the most accurate tracking can be achieved by combining multiple trackers.

Web-style ranking and SLU combination for dialog state tracking

“…In this paper we use data and evaluation metrics from the second dialog state tracking challenge (DSTC2) (Henderson et al, 2014;Henderson et al, 2013a). Dialogs in DSTC2 are in the restaurant search domain.…”

The Dialog State Tracking Challenge Series

“…Participants have mostly been academic instutions, with a minority of corporate research labs. Results have been presented at special sessions: DSTC1 at the annual Special Interest Group on Discourse and Dialogue (SIGdial) conference in 2013 (Williams et al 2013); DSTC2 at SIGdial in June 2014 (Henderson, Thomson, and Williams 2014); and DSTC3 at IEEE Spoken Language Technologies (SLT) Workshop in December 2014 (forthcoming).…”

Extrinsic Evaluation of Dialog State Tracking and Predictive Metrics for Dialog Policy Optimization

“…Also, since the use of an approximate policy evaluation method (e.g. LSTD) can introduce systemic errors, more deliberate experimental setups will be designed for a future study: 1) the application of different RL algorithms for training and test datasets 2) further experiments on different datasets, e.g., the datasets for DSTC2 (Henderson et al, 2014). Although the state representation adopted in this work is quite common for most systems that use a POMDP model, different state representations could possibly reveal new insights.…”