Development and Evaluation of ClientBot: Patient-Like Conversational Agent to Train Basic Counseling Skills

Abstract

Background Training therapists is both expensive and time-consuming. Degree–based training can require tens of thousands of dollars and hundreds of hours of expert instruction. Counseling skills practice often involves role-plays, standardized patients, or practice with real clients. Performance–based feedback is critical for skill development and expertise, but trainee therapists often receive minimal and subjective feedback, which is distal to their skill practice. Objective In this study, we developed and evaluated a patient-like neural conversational agent, which provides real-time feedback to trainees via chat–based interaction. Methods The text–based conversational agent was trained on an archive of 2354 psychotherapy transcripts and provided specific feedback on the use of basic interviewing and counseling skills (ie, open questions and reflections—summary statements of what a client has said). A total of 151 nontherapists were randomized to either (1) immediate feedback on their use of open questions and reflections during practice session with ClientBot or (2) initial education and encouragement on the skills. Results Participants in the ClientBot condition used 91% (21.4/11.2) more reflections during practice with feedback ( P <.001) and 76% (14.1/8) more reflections after feedback was removed ( P <.001) relative to the control group. The treatment group used more open questions during training but not after feedback was removed, suggesting that certain skills may not improve with performance–based feedback. Finally, after feedback was removed, the ClientBot group used 31% (32.5/24.7) more listening skills overall ( P <.001). Conclusions This proof-of-concept study demonstrates that practice and feedback can improve trainee use of basic counseling skills.

A Physical Activity and Diet Program Delivered by Artificially Intelligent Virtual Health Coach: Proof-of-Concept Study

“…Similar to the previous review, most recent studies were aimed at mental health self-management (n=5) [ 10 - 12 ]. Single studies (n=3) used virtual assistants to provide patients with education and support for breast cancer [ 13 ], genetic counseling [ 14 ], and clinician training [ 15 ]. Despite the potential for natural language virtual assistants to provide personalized information and support users to engage in positive health behaviors, only three natural language processing virtual assistants focus on lifestyle modification.…”

Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support

“…Actionable Feedback. Current AI-in-the-loop systems are often limited to addressing "what" (rather than "how") participants should improve [52][53][54][55] . For such a goal, it is generally acceptable to design simple interfaces that prompt participants to leverage strategies for successful supportive conversations (e.g., prompting "you may want to empathize with the user") without any instructions on how to concretely apply those strategies.…”

“…This design reflects the high-risk setting of mental health, where AI is likely best used to augment, rather than replace, human skills 46,51 . Furthermore, while current AI-in-the-loop systems are often restricted in the extent to which they can guide humans (e.g., simple classification methods that tell users to be empathic when they are not) [52][53][54][55] , we ensure actionability by guiding peer supporters with concrete steps they may take to respond with more empathy, e.g., through the insertion of new empathic sentences or replacement of existing low-empathy sentences with their more empathic counterparts (Figure 1c). For complex, hard-to-learn skills like empathy, this enables just-in-time suggestions on not just "what" to improve but on "how" to improve it.…”

Improving Mental Health Services: A 50-Year Journey from Randomized Experiments to Artificial Intelligence and Precision Mental Health

“…Another example of the use of NLP application is the use of a bot that was trained to assess and provide feedback on specific interviewing and counseling skills such as asking open-ended questions and providing feedback (Tanana et al 2019 ). After training the bot on 2345 transcripts, 151 non-therapists (using Amazon Mechanical Turk recruits) were randomly assigned to either immediate feedback on a practice session with the bot or just encouragement on the use of those skills.…”