The Power of Prompt Tuning for Low-Resource Semantic Parsing

Schucher, Nathan; Reddy, Siva; Vries, Harm de

doi:10.18653/v1/2022.acl-short.17

Cited by 16 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Existing prompt tuning methods attempt to find the optimal prompts after adding prompts to the input to adapt a fixed language model to a specific task. However, the performance of prompt tuning depends primarily on the frequency of the terms in the pretraining corpus [16]. We introduce a domain-specific prompt tuning method that adapts the frozen language model to data distribution in the downstream domain to address this issue.…”

Section: Methodsmentioning

confidence: 99%

Multi-Stage Prompt Tuning for Political Perspective Detection in Low-Resource Settings

Kim,

Lee,

Won

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

Political perspective detection in news media—identifying political bias in news articles—is an essential but challenging low-resource task. Prompt-based learning (i.e., discrete prompting and prompt tuning) achieves promising results in low-resource scenarios by adapting a pre-trained model to handle new tasks. However, these approaches suffer performance degradation when the target task involves a textual domain (e.g., a political domain) different from the pre-training task (e.g., masked language modeling on a general corpus). In this paper, we develop a novel multi-stage prompt tuning framework for political perspective detection. Our method involves two sequential stages: a domain- and task-specific prompt tuning stage. In the first stage, we tune the domain-specific prompts based on a masked political phrase prediction (MP3) task to adjust the language model to the political domain. In the second task-specific prompt tuning stage, we only tune task-specific prompts with a frozen language model and domain-specific prompts for downstream tasks. The experimental results demonstrate that our method significantly outperforms fine-tuning (i.e., model tuning) methods and state-of-the-art prompt tuning methods on the SemEval-2019 Task 4: Hyperpartisan News Detection and AllSides datasets.

show abstract

Section: Methodsmentioning

confidence: 99%

Multi-Stage Prompt Tuning for Political Perspective Detection in Low-Resource Settings

Kim,

Lee,

Won

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…LLMs are increasingly used for semantic parsing in low-data scenarios utilizing canonical representations (Shin et al, 2021;Yang et al, 2022), and prompt-tuning (Schucher et al, 2022;Drozdov et al, 2022). The closest work to ours is Zhao et al (2022) where they decompose parsing into QA tasks.…”

Section: Related Workmentioning

confidence: 99%

ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models

Mekala,

Wolfe,

Roy

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

We explore the use of large language models (LLMs) for zero-shot semantic parsing. Semantic parsing involves mapping natural language utterances to task-specific meaning representations. LLMs are generally trained on publicly available text and code and cannot be expected to directly generalize to domain-specific parsing tasks in a zero-shot setting. In this work, we propose ZEROTOP, a zero-shot task-oriented parsing method that decomposes semantic parsing problem into a set of abstractive and extractive question-answering (QA) problems. For each utterance, we prompt the LLM with questions corresponding to its top-level intent and a set of slots and use the LLM generations to construct the target meaning representation. We observe that current LLMs fail to detect unanswerable questions; and as a result, cannot handle questions corresponding to missing slots. We address this by fine-tuning a language model on public QA datasets using synthetic negative samples. Experimental results show that our QA-based decomposition paired with the fine-tuned LLM can zero-shot parse ≈ 16% of utterances in the MTOP dataset.* Work done during an internship at Microsoft Semantic Machines.♠ Work done while at Microsoft Semantic Machines.

show abstract

“…Our paper tackles multi-aspect few-shot learning where the target is a structured output containing multiple attributes, and the model only has a few examples from which to learn such a pattern. Given the complexity of the task, previous research trained custom models for each dataset (Zheng et al, 2022;Schucher et al, 2022). Instead, we leverage the power of LLMs (Brown et al, 2020;Chung et al, 2022) to design a more general solution through controlled data synthesis.…”

Section: Related Workmentioning

confidence: 99%

Mixture of Soft Prompts for Controllable Data Generation

Chen,

Lee,

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Large language models (LLMs) effectively generate fluent text when the target output follows natural language patterns. However, structured prediction tasks confine the output format to a limited ontology, causing even very large models to struggle since they were never trained with such restrictions in mind. The difficulty of using LLMs for direct prediction is exacerbated in few-shot learning scenarios, which commonly arise due to domain shift and resource limitations. We flip the problem on its head by leveraging the LLM as a tool for data augmentation rather than a model for direct prediction. Our proposed Mixture of Soft Prompts (MSP) serves as a parameter-efficient procedure for generating multi-attribute data in a controlled manner. Denoising mechanisms are further applied to improve the quality of synthesized data. Automatic metrics show our method is capable of producing diverse and natural text, while preserving label semantics. Moreover, MSP achieves state-of-the-art results on three benchmarks when compared against strong baselines. Our method offers an alternate data-centric approach for applying LLMs to complex prediction tasks.

show abstract

The Power of Prompt Tuning for Low-Resource Semantic Parsing

Cited by 16 publications

References 24 publications

Multi-Stage Prompt Tuning for Political Perspective Detection in Low-Resource Settings

Multi-Stage Prompt Tuning for Political Perspective Detection in Low-Resource Settings

ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models

Mixture of Soft Prompts for Controllable Data Generation

Contact Info

Product

Resources

About