A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement

Sarkar, Surjodeep; Gaur, Manas; Chen, Lujie Karen; Garg, Muskan; Srivastava, Biplav

doi:10.3389/frai.2023.1229805

Cited by 8 publications

(6 citation statements)

References 120 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This analysis serves to illustrate the current shortcomings in effectively addressing potentially hazardous mental states, deficiencies that could lead to severe consequences if these systems are deployed without due consideration in the context of mental health support. While conversational AI exhibits promising capabilities, compelling evidence indicates that it may advance at a pace that outstrips associated safety measures [ 28 ]. While the rapid progress in innovation is undeniably fascinating, it is imperative for researchers and developers to proactively prioritize ethical considerations pertaining to transparency, explainability, bias mitigation, user privacy, system access controls, and the micro-targeting of vulnerable populations.…”

Section: Discussionmentioning

confidence: 99%

Safety of Large Language Models in Addressing Depression

Heston

2023

Cureus

View full text Add to dashboard Cite

Background Generative artificial intelligence (AI) models, exemplified by systems such as ChatGPT, Bard, and Anthropic, are currently under intense investigation for their potential to address existing gaps in mental health support. One implementation of these large language models involves the development of mental health-focused conversational agents, which utilize pre-structured prompts to facilitate user interaction without requiring specialized knowledge in prompt engineering. However, uncertainties persist regarding the safety and efficacy of these agents in recognizing severe depression and suicidal tendencies. Given the well-established correlation between the severity of depression and the risk of suicide, improperly calibrated conversational agents may inadequately identify and respond to crises. Consequently, it is crucial to investigate whether publicly accessible repositories of mental health-focused conversational agents can consistently and safely address crisis scenarios before considering their adoption in clinical settings. This study assesses the safety of publicly available ChatGPT-3.5 conversational agents by evaluating their responses to a patient simulation indicating worsening depression and suicidality. Methodology This study evaluated ChatGPT-3.5 conversational agents on a publicly available repository specifically designed for mental health counseling. Each conversational agent was evaluated twice by a highly structured patient simulation. First, the simulation indicated escalating suicide risk based on the Patient Health Questionnaire (PHQ-9). For the second patient simulation, the escalating risk was presented in a more generalized manner not associated with an existing risk scale to assess the more generalized ability of the conversational agent to recognize suicidality. Each simulation recorded the exact point at which the conversational agent recommended human support. Then, the simulation continued until the conversational agent stopped entirely and shut down completely, insisting on human intervention. Results All 25 agents available on the public repository FlowGPT.com were evaluated. The point at which the conversational agents referred to a human occurred around the mid-point of the simulation, and definitive shutdown predominantly only happened at the highest risk levels. For the PHQ-9 simulation, the average initial referral and shutdown aligned with PHQ-9 scores of 12 (moderate depression) and 25 (severe depression). Few agents included crisis resources - only two referenced suicide hotlines. Despite the conversational agents insisting on human intervention, 22 out of 25 agents would eventually resume the dialogue if the simulation reverted to a lower risk level. Conclusions Current generative AI-based conversational agents are slow to escalate mental health risk scenarios, postponing referral to a human to potentially dangerous levels. More rigorous testing and oversig...

show abstract

Section: Discussionmentioning

confidence: 99%

Safety of Large Language Models in Addressing Depression

Heston

2023

Cureus

View full text Add to dashboard Cite

show abstract

“…Such attributes are important when LLMs are designed for critical applications like motivational interviewing (Sarkar et al. 2023). Motivational interviewing is a communication style often used in mental health counseling, and ensuring logical coherence and semantic relatedness in generated responses is crucial for effective interactions (Shah et al.…”

Section: Defining Consistency Reliability User‐level Explainability A...mentioning

confidence: 99%

Building trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety

Gaur,

Sheth

2024

AI Magazine

Self Cite

View full text Add to dashboard Cite

Explainability and Safety engender trust. These require a model to exhibit consistency and reliability. To achieve these, it is necessary to use and analyze data and knowledge with statistical and symbolic AI methods relevant to the AI application––neither alone will do. Consequently, we argue and seek to demonstrate that the NeuroSymbolic AI approach is better suited for making AI a trusted AI system. We present the CREST framework that shows how Consistency, Reliability, user‐level Explainability, and Safety are built on NeuroSymbolic methods that use data and knowledge to support requirements for critical applications such as health and well‐being. This article focuses on Large Language Models (LLMs) as the chosen AI system within the CREST framework. LLMs have garnered substantial attention from researchers due to their versatility in handling a broad array of natural language processing (NLP) scenarios. As examples, ChatGPT and Google's MedPaLM have emerged as highly promising platforms for providing information in general and health‐related queries, respectively. Nevertheless, these models remain black boxes despite incorporating human feedback and instruction‐guided tuning. For instance, ChatGPT can generate unsafe responses despite instituting safety guardrails. CREST presents a plausible approach harnessing procedural and graph‐based knowledge within a NeuroSymbolic framework to shed light on the challenges associated with LLMs.

show abstract

“…In psychiatry, LLMs like ChatGPT can provide accessible mental health services, breaking down geographical, financial, or temporal barriers, which are particularly pronounced in mental health care (4,9). For instance, ChatGPT can support therapists by offering tailored assistance during various treatment phases, from initial assessment to post-treatment recovery (10)(11)(12). This includes aiding in symptom management and encouraging healthy lifestyle changes pertinent to psychiatric care (10)(11)(12)(13)(14).…”

Section: Introductionmentioning

confidence: 99%

“…For instance, ChatGPT can support therapists by offering tailored assistance during various treatment phases, from initial assessment to post-treatment recovery (10)(11)(12). This includes aiding in symptom management and encouraging healthy lifestyle changes pertinent to psychiatric care (10)(11)(12)(13)(14).…”

Section: Introductionmentioning

confidence: 99%

Applications of Large Language Models in Psychiatry: A Systematic Review

Omar,

Soffer,

Charney

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: With their unmatched ability to interpret and engage with human language and context, large language models (LLMs) hint at the potential to bridge AI and human cognitive processes. This review explores the current application of LLMs, such as ChatGPT, in the field of psychiatry. Methods: We followed PRISMA guidelines and searched through PubMed, Embase, Web of Science, and Scopus, up until March 2024. Results: From 771 retrieved articles, we included 16 that directly examine LLMs' use in psychiatry. LLMs, particularly ChatGPT and GPT-4, showed diverse applications in clinical reasoning, social media, and education within psychiatry. They can assist in diagnosing mental health issues, managing depression, evaluating suicide risk, and supporting education in the field. However, our review also points out their limitations, such as difficulties with complex cases and potential underestimation of suicide risks. Conclusion: Early research in psychiatry reveals LLMs' versatile applications, from diagnostic support to educational roles. Given the rapid pace of advancement, future investigations are poised to explore the extent to which these models might redefine traditional roles in mental health care.

show abstract

A review of the explainability and safety of conversational agents for mental health to identify avenues for improvement

Cited by 8 publications

References 120 publications

Safety of Large Language Models in Addressing Depression

Safety of Large Language Models in Addressing Depression

Building trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety

Applications of Large Language Models in Psychiatry: A Systematic Review

Contact Info

Product

Resources

About