A Multi-Persona Chatbot for Hotline Counselor Training

Abstract

Suicide prevention hotline counselors aid individuals during difficult times through millions of calls and chats. A chatbot cannot safely replace a counselor, but we explore whether a chatbot can be developed to help train human counselors. Such a system needs to simulate intimate situations across multiple practice sessions. Open-domain dialogue systems frequently suffer from generic responses that do not characterize personal stories, so we look to infuse conversations with persona information by mimicking prototype conversations. Towards building a "Crisisbot" hotline visitor simulation, we propose a counseling strategy annotation scheme and a multi-task framework that leverages these counselor strategies to retrieve similar examples, generate diverse subutterances, and interleave prototype and generated sub-utterances into complex responses. We evaluate this framework with crowdworkers and experienced hotline counselors. The framework considerably increases response diversity and specificity, with limited impact to coherence. Our results also show a considerable discrepancy between crowdworker and counselor judgements, which emphasizes the importance of including target populations in system development and evaluation.

Introducing CounseLLMe: A dataset of simulated mental health dialogues for comparing LLMs like Haiku, LLaMAntino and ChatGPT against humans

“…Although virtual patients can be easy to develop, "realistic" virtual patients are way more complex. Some attempts [31] have been made using traditional machine learning techniques like LSTMs (Long Short-Term Memory neural networks [32]) and some companies have already deployed tools to help train practitioners in mental health (Link: https://www.lyssn.io/, Last Accessed: 17/04/2024). However, to the best of our knowledge, [33] was the first to use an LLM model to simulate a virtual patient.…”

A Framework for Exploring Player Perceptions of LLM-Generated Dialogue in Commercial Video Games

“…Rather than upend these familiar techniques for more flexible approaches which have yet to gain traction (Riedl and Young, 2004;Mateas and Stern, 2005), likely due to their complexity, we opt to augment the existing approaches actively in use in commericial video games. Our work bridges the tightly scripted scenarios common to video games, with more natural speech that offers humans more agency over their interaction with virtual agents, leading to easier to design agent interactions useful for training simulations (Demasi et al, 2020) and tutoring (Wang et al, 2023a).…”

Comparing Experts and Novices for AI Data Work: Insights on Allocating Human Intelligence to Design a Conversational Agent

“…Results showed that responses created by experts are significantly longer (p-value < 1e-3) than novices as experts' responses contain 18.6 words on average while novices' responses contain 9.6 words on average. To further assess the quality of new responses created by experts and novices, we coded the coherence (Demasi, Li, and Yu 2020) of the response as well as whether or not the response contained emotional disclosure and/or new story information(Chen, Wu, and Yang 2020) as described in Section . To compare the frequency of each code by each participant, we built a linear regression model for each code.…”