PLACES: Prompting Language Models for Social Conversation Synthesis

Abstract

Collecting high quality conversational data can be very expensive for most applications and infeasible for others due to privacy, ethical, or similar concerns. A promising direction to tackle this problem is to generate synthetic dialogues by prompting large language models. In this work, we use a small set of expertwritten conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations. This includes various dimensions of conversation quality with human evaluation directly on the synthesized conversations, and interactive human evaluation of chatbots fine-tuned on the synthetically generated dataset. We additionally demonstrate that this prompting approach is generalizable to multi-party conversations, providing potential to create new synthetic data for multi-party tasks. Our synthetic multi-party conversations were rated more favorably across all measured dimensions compared to conversation excerpts sampled from a human-collected multi-party dataset. Random In-Context Examples Target Recipe <Conversation 0> The following is a conversation between Alice and Bob about relationships. Bob recently got engaged. Alice: Congrats on your engagement! When do you think you will have your wedding? Bob: Thank you!! We're thinking of having it in November. Alice: That's amazing! Will you pick a fancy destination? ... <Conversation 2> <Conversation 3> The following is a conversation between Alice and Bob about pets. Alice likes cats but Bob likes dogs. Result: Alice: What kind of pet do you have? Bob: I have a dog. Her name is Hope. Alice: Are you an animal person? Bob: I do love animals. I think they're so special. Alice: I just adopted a kitten from a shelter. Her name is Mimi. Bob: Aww, I hope she's doing well. You must be a really good pet parent. Alice: Thanks! I just wanted to do what I could to make sure she was safe. Bob: That's the greatest thing anyone can do for a pet. Well, I should probably go now. Alice: OK! Have a good night.

NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation

“…These conversations contain dialogues and chitchat from sources such as TV shows, Vlogs, and other types of videos from Bilibili, a Chinese video-sharing platform. For English dialogues, as no existing domain-comparable dialogue is available, we compare against a set of 200 realistic, human-written dialogues reflecting daily communication that covers various topics about daily life, DailyDialog (Li et al, 2017); a comparison that has been used for previous evaluations of synthetic dialogue quality (Chen et al, 2023).…”

“…LLMs for Synthetic Data Generation. Prompting LLMs to synthesize and augment language data for existing tasks (Li et al, 2022;Møller et al, 2023;Chen et al, 2023) has emerged as a viable, cost-effective alternative in lieu of crowd-sourced annotation at scale or alternative strategies such as fine-tuning language generators (Papangelis et al, 2021;Zhang et al, 2020) in the dialogue domain. LLMs, trained on massive amounts of web text, suffer from representational and allocational harms (Blodgett et al, 2020;Weidinger et al, 2021).…”

“…As a small step towards addressing this gap, leveraging recent successes in utilizing large language models (LLMs) for social data generation and augmentation (Kim et al, 2022a;Chen et al, 2023), we propose a human-in-the-loop framework to synthesize realistic conversational data under expert prompting for modeling social norm adher-ence and violation. Using this human-AI collaboration framework, we generate a series of 4231 dyadic dialogues totaling 29550 conversational turns grounded in theoretical norm categories (Linguistic Data Consortium, 2022) across Chinese and American cultures, and demonstrate that our synthetic bilingual conversations are comparable to or exceed the quality of existing, naturally occurring datasets under interactive human evaluation and automatic metrics; examples of our dialogues are shown in Figure 1.…”