Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Sahu, Gaurav; Rodríguez, Paul; Laradji, Issam; Atighehchian, Parmida; Vázquez, David; Bahdanau, Dzmitry

doi:10.48550/arxiv.2204.01959

“…PLMs have demonstrated impressive emerging conversational capabilities, enabling big performance improvements in various dialogue tasks (Brown et al, 2020;Shuster et al, 2022;Peng et al, 2022;Kulhánek et al, 2021). Particularly, PLMs have been prompted to augment existing conversational data (Chen et al, 2022;Mehri et al, 2022;Sahu et al, 2022).…”

Section: Triadic Conversations Dyadic Conversationsmentioning

confidence: 99%

“…In particular, in-context learning, where few-shot examples are provided in the input prompt of a PLM, has been found to provide valuable information in guiding generation output (Min et al, 2022;Brown et al, 2020;Min et al, 2021;Lu et al, 2021b). As a result, many recent efforts in prompting PLMs have sought to augment various natural language processing datasets (Chen et al, 2022;Sahu et al, 2022;Mehri et al, 2022;Rosenbaum et al, 2022a). Prompting has become a viable "solution" for augmentation in dialogue tasks, which have traditionally been considered challenging due to the difficulty of augmenting dialogue context (Chen et al, 2022).…”

Section: Related Workmentioning

confidence: 99%

“…However, prompt-based augmentation strategies are uncontrolled forms of generation, which may result in generation mistakes for labeled datasets (Sahu et al, 2022;Chen et al, 2022;Meng et al, 2022). In contrast, other recent studies have instead proposed language augmentation strategies that use complex, highly-controlled frameworks that often involve fine-tuning generators (Papangelis et al, 2021;Kulhánek et al, 2021;.…”

Section: Related Workmentioning

confidence: 99%

“…Our work differs from existing applications of prompting for conversations along several dimensions. Many studies examine utterance-level generation (Chen et al, 2022;Sahu et al, 2022;Aher et al, 2022;Rosenbaum et al, 2022b), whereas our work concerns the synthesis of full conversations. Bae et al (2022) generated conversations for a narrow task and provided evaluations between their synthesis conditions.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

PLACES: Prompting Language Models for Social Conversation Synthesis

Chen¹,

Papangelis²,

Tao³

et al. 2023

Preprint

0

View full text Add to dashboard Cite

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expert-written conversations as incontext examples to synthesize a social conversation dataset using prompting 1 . We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multiparty conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset.

show abstract

“…PLMs have demonstrated impressive emerging conversational capabilities, enabling big performance improvements in various dialogue tasks (Brown et al, 2020;Shuster et al, 2022;Peng et al, 2022;Kulhánek et al, 2021). Particularly, PLMs have been prompted to augment existing conversational data (Chen et al, 2022;Mehri et al, 2022;Sahu et al, 2022).…”

Section: Triadic Conversations Dyadic Conversationsmentioning

confidence: 99%

PLACES: Prompting Language Models for Social Conversation Synthesis

Chen,

Papangelis,

Tao

et al. 2023

Findings of the Association for Computational Linguistics: EACL 2023

2

0

View full text Add to dashboard Cite

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expertwritten conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset. Random In-Context Examples Target Recipe The following is a conversation between Alice and Bob about relationships. Bob recently got engaged. Alice: Congrats on your engagement! When do you think you will have your wedding? Bob: Thank you!! We're thinking of having it in November. Alice: That's amazing! Will you pick a fancy destination? ... The following is a conversation between Alice and Bob about pets. Alice likes cats but Bob likes dogs. Result: Alice: What kind of pet do you have? Bob: I have a dog. Her name is Hope. Alice: Are you an animal person? Bob: I do love animals. I think they're so special. Alice: I just adopted a kitten from a shelter. Her name is Mimi. Bob: Aww, I hope she's doing well. You must be a really good pet parent. Alice: Thanks! I just wanted to do what I could to make sure she was safe. Bob: That's the greatest thing anyone can do for a pet. Well, I should probably go now. Alice: OK! Have a good night.

show abstract

ASMR: Aggregated Semantic Matching Retrieval Unleashing Commonsense Ability of LLM through Open-Ended Question Answering

Lin,

Chandra,

Hsu

2024

AAAI-SS

0

View full text Add to dashboard Cite

Commonsense reasoning refers to the ability to make inferences, draw conclusions, and understand the world based on general knowledge and commonsense. Whether Large Language Models (LLMs) have commonsense reasoning ability remains a topic of debate among researchers and experts. When confronted with multiple-choice commonsense reasoning tasks, humans typically rely on their prior knowledge and commonsense to formulate a preliminary answer in mind. Subsequently, they compare this preliminary answer to the provided choices, and select the most likely choice as the final answer. We introduce Aggregated Semantic Matching Retrieval (ASMR) as a solution for multiple-choice commonsense reasoning tasks. To mimic the process of humans solving commonsense reasoning tasks with multiple choices, we leverage the capabilities of LLMs to first generate the preliminary possible answers through open-ended question which aids in enhancing the process of retrieving relevant answers to the question from the given choices. Our experiments demonstrate the effectiveness of ASMR on popular commonsense reasoning benchmark datasets, including CSQA, SIQA, and ARC (Easy and Challenge). ASMR achieves state-of-the-art (SOTA) performance with a peak of +15.3% accuracy improvement over the previous SOTA on SIQA dataset.

show abstract

Data Augmentation for Intent Classification with Off-the-shelf Large Language Models

Cited by 5 publications

References 13 publications

PLACES: Prompting Language Models for Social Conversation Synthesis

PLACES: Prompting Language Models for Social Conversation Synthesis

PLACES: Prompting Language Models for Social Conversation Synthesis

ASMR: Aggregated Semantic Matching Retrieval Unleashing Commonsense Ability of LLM through Open-Ended Question Answering

Contact Info

Product

Resources

About