Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking

Abstract

Annotating task-oriented dialogues is notorious for the expensive and difficult data collection process. Few-shot dialogue state tracking (DST) is a realistic solution to this problem. In this paper, we hypothesize that dialogue summaries are essentially unstructured dialogue states; hence, we propose to reformulate dialogue state tracking as a dialogue summarization problem. To elaborate, we train a text-totext language model with synthetic templatebased dialogue summaries, generated by a set of rules. Then, the dialogue states can be recovered by inversely applying the summary generation rules. We empirically show that our method DS2 outperforms previous works on few-shot DST in MultiWoZ 2.0 and 2.1, in both cross-domain and multi-domain settings. Our method 1 also exhibits vast speedup during both training and inference as it can generate all states at once. Finally, based on our analysis, we discover that the naturalness of the summary templates plays a key role for successful training.

Introduction: sustainable Logistics Systems using AI-based Meta-Heuristics Approaches

“…To address zero-shot DST in unseen domains, previous cross-domain transfer strategies based on small models typically leverage extra dialogue corpora in similar domains Lin et al, 2021b;Su et al, 2021) or redefining DST in terms of other types of tasks, such as question answering (Lin et al, 2021c) or summarization (Shin et al, 2022) to find appropriate additional training data. Despite these efforts, their overall zero-shot performance remains relatively low.…”

DiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning

“…Another set of approaches aim to improve zeroshot performance by exploiting external knowledge and datasets from other natural language tasks before fine-tuning a model for DST. For instance, Gao et al (2020); Li et al (2021); Lin et al (2021a) pre-train models on reading comprehension data, Shin et al (2022) reformulate DST as a dialogue summarization task with external annotated data, and Hudeček et al (2021) use semantic analysis and named entity recognition to identify slots. In contrast, our approach does not require any extra datasets or training efforts .…”

“…DS2 (Shin et al, 2022) treats DST as a dialogue summarization task, and fine-tunes T5-large and BART models with synthetic summary templates. (Hu et al, 2022) reformulates DST as a text-to-SQL task and transforms relevant in-context examples to SQL queries and prompts a Codex model without any fine-tuning.…”

“…We note that our retriever does not result in any information leakage since no examples and slots are included from the unseen domain. Cross-domain few-shot -We include three fewshot settings, where the retriever and model additionally have access to 1%, 5%, and 10% of training data from the unseen domain, similar to Shin et al (2022); Wu et al (2019). Multi-domain few-shot -We follow the multidomain scenario from Shin et al (2022); Wu et al (2020), where 1%, 5%, and 10% of the entire training data are sampled for model training and retrieval.…”

“…Cross-domain few-shot -We include three fewshot settings, where the retriever and model additionally have access to 1%, 5%, and 10% of training data from the unseen domain, similar to Shin et al (2022); Wu et al (2019). Multi-domain few-shot -We follow the multidomain scenario from Shin et al (2022); Wu et al (2020), where 1%, 5%, and 10% of the entire training data are sampled for model training and retrieval. Note: We do not include the zero-shot results from prior in-context learning work (Hu et al, 2022) (IC-DST) since their prompt examples are designed to include information from the 'unseen' domain, which results in information leakage to the model, and hence does not reflect the traditional zero-shot learning setting.…”