Dialogue State Tracking with a Language Model using Schema-Driven Prompting

Lee, Chia‐Hsuan; Cheng, Hui‐Ming; Ostendorf, Mari

doi:10.48550/arxiv.2109.07506

Cited by 5 publications

(9 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also evaluate the performance of LDST using the complete training data, and compare it with the following strong baselines, including SGDbaseline , TRADE , DS-DST (Zhang et al, 2019), TripPy (Heck et al, 2020), Seq2Seq-DU (Feng et al, 2020), MetaASSIST (Ye et al, 2022b), SDP-DST (Lee et al, 2021), TOATOD (Bang et al, 2023b), DiCoS-DST (Guo et al, 2022b), D3ST (Zhao et al, 2022), paDST (Ma et al, 2019). And the results are shown on Table 5.…”

Section: Results Of Fine-tuning With Full Training Datamentioning

confidence: 99%

“…Recent advancements in parameter-efficient fine-tuning (PEFT) techniques have effectively alleviated this problem, such as LoRA (Hu et al, 2021) and Prefix Tuning (Liu et al, 2021). For instance, both Lee et al (2021) and proposed a prompt-tuning method that leverages domain-specific prompts and context information to improve the performance of DST task. Meanwhile, Ma et al (2023) and introduced the prefix tuning approach, which involves modifying the input prompt by adding specific tokens at the beginning of the dialogue, aiming to enhance the efficiency of model fine-tuning.…”

Section: Llms For Dialogue State Trackingmentioning

confidence: 99%

“…Step 1 -Standard Preprocessing In line with the approach used by Lee et al (2021) In this example, the "dialogue" field includes the content of the dialogue (A1, U 1), (A2, U 2), the tracked slot <restaurant-area>, and it's description. The "state" field is the value of the corresponding slot.…”

Section: C1 Data Preprocessing and Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Towards LLM-driven Dialogue State Tracking

Feng,

Lu,

Liu

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Dialogue State Tracking (DST) is of paramount importance in ensuring accurate tracking of user goals and system actions within taskoriented dialogue systems. The emergence of large language models (LLMs) such as GPT3 and ChatGPT has sparked considerable interest in assessing their efficacy across diverse applications. In this study, we conduct an initial examination of ChatGPT's capabilities in DST. Our evaluation uncovers the exceptional performance of ChatGPT in this task, offering valuable insights to researchers regarding its capabilities and providing useful directions for designing and enhancing dialogue systems. Despite its impressive performance, ChatGPT has significant limitations including its closedsource nature, request restrictions, raising data privacy concerns, and lacking local deployment capabilities. To address these concerns, we present LDST, an LLM-driven DST framework based on smaller, open-source foundation models. By utilizing a novel domain-slot instruction tuning method, LDST achieves performance on par with ChatGPT. Comprehensive evaluations across three distinct experimental settings, we find that LDST exhibits remarkable performance improvements in both zero-shot and few-shot setting compared to previous SOTA methods. The source code 1 is provided for reproducibility.

show abstract

Section: Results Of Fine-tuning With Full Training Datamentioning

confidence: 99%

Section: Llms For Dialogue State Trackingmentioning

confidence: 99%

See 1 more Smart Citation

Towards LLM-driven Dialogue State Tracking

Feng,

Lu,

Liu

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…However, evidence collection systems proposed by prior studies [Minhas et al(2022), Ku et al(2008)] either rely on rule-based methods or require a large amount of manually labeled training data. Future studies can adopt methods from recent NLP works from the domain of dialogue state tracking , Gao et al(2019), Lee et al(2021)] and event argument extraction [Du and Cardie(2020), Lyu et al(2021)]. We also suggest future system design practitioners consider using pre-trained language models for more effective and robust event information extraction, by increasing the model's ability in domain adaptation and commonsense reasoning [Yang et al(2019)].…”

Section: Designing Conversational Agents For High-quality Reportingmentioning

confidence: 99%

Discovering the Hidden Facts of User-Dispatcher Interactions via Text-based Reporting Systems for Community Safety

Liu¹,

Mayfield²,

Huang³

2022

Preprint

View full text Add to dashboard Cite

Recently, an increasing number of safety organizations in the U.S. have incorporated text-based risk reporting systems to respond to safety incident reports from their community members. To gain a better understanding of the interaction between community members and dispatchers using text-based risk reporting systems, this study conducts a system log analysis of LiveSafe, a community safety reporting system, to provide empirical evidence of the conversational patterns between users and dispatchers using both quantitative and qualitative methods. We created an ontology to capture information (e.g., location, attacker, target, weapon, start-time, and end-time, etc.) that dispatchers often collected from users regarding their incident tips. Applying the proposed ontology, we found that dispatchers often asked users for different information across varied event types (e.g., Attacker for Abuse and Attack events, Target for Harassment events). Additionally, using emotion detection and regression analysis, we found an inconsistency in dispatchers' emotional support and responsiveness to users' messages between different organizations and between incident categories. The results also showed that users had a higher response rate and responded quicker when dispatchers provided emotional support. These novel findings brought significant insights to both practitioners and system designers, e.g., AI-based solutions to augment human agents' skills for improved service quality.

show abstract

“…Dialogue State Tracking (DST) is crucial in Task-Oriented Dialogue (TOD) systems to understand and manage user intentions (Wu et al, 2019;Hosseini-Asl et al, 2020;Heck et al, 2020;Lee et al, 2021;Zhao et al, 2022). Collecting and annotating dialogue states at the turn level is challenging and expensive (Budzianowski et al, 2018), and commercial applications often need to expand the schema and include new domains.…”

Section: Introductionmentioning

confidence: 99%

Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking

Wu,

Dong,

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Zero-shot Dialogue State Tracking (DST) addresses the challenge of acquiring and annotating task-oriented dialogues, which can be timeconsuming and costly. However, DST extends beyond simple slot-filling and requires effective updating strategies for tracking dialogue state as conversations progress. In this paper, we propose ParsingDST, a new In-Context Learning (ICL) method, to introduce additional intricate updating strategies in zero-shot DST. Our approach reformulates the DST task by leveraging powerful Large Language Models (LLMs) and translating the original dialogue text to JSON through semantic parsing as an intermediate state. We also design a novel framework that includes more modules to ensure the effectiveness of updating strategies in the text-to-JSON process. Experimental results demonstrate that our approach outperforms existing zero-shot DST methods on MultiWOZ, exhibiting significant improvements in Joint Goal Accuracy (JGA) and slot accuracy compared to existing ICL methods.

show abstract

Dialogue State Tracking with a Language Model using Schema-Driven Prompting

Cited by 5 publications

References 25 publications

Towards LLM-driven Dialogue State Tracking

Towards LLM-driven Dialogue State Tracking

Discovering the Hidden Facts of User-Dispatcher Interactions via Text-based Reporting Systems for Community Safety

Semantic Parsing by Large Language Models for Intricate Updating Strategies of Zero-Shot Dialogue State Tracking

Contact Info

Product

Resources

About