Towards Emotion- and Time-Aware Classification of Tweets to Assist Human Moderation for Suicide Prevention

Abstract

Social media platforms are already engaged in leveraging existing online socio-technical systems to employ just-in-time interventions for suicide prevention to the public. These efforts primarily rely on self-reports of potential self-harm content that is reviewed by moderators. Most recently, platforms have employed automated models to identify self-harm content, but acknowledge that these automated models still struggle to understand the nuance of human language (e.g., sarcasm). By explicitly focusing on Twitter posts that could easily be misidentified by a model as expressing suicidal intent (i.e., they contain similar phrases such as ``wanting to die''), our work examines the temporal differences in historical expressions of general and emotional language prior to a clear expression of suicidal intent. Additionally, we analyze time-aware neural models that build on these language variants and factors in the historical, emotional spectrum of a user's tweeting activity. The strongest model achieves high (statistically significant) performance (macro F1=0.804, recall=0.813) to identify social media indicative of suicidal intent. Using three use cases of tweets with phrases common to suicidal intent, we qualitatively analyze and interpret how such models decided if suicidal intent was present and discuss how these analyses may be used to alleviate the burden on human moderators within the known constraints of how moderation is performed (e.g., no access to the user's timeline). Finally, we discuss the ethical implications of such data-driven models and inferences about suicidal intent from social media. Content warning: this article discusses self-harm and suicide.

Applications of Large Language Models in the Field of Suicide Prevention: A Scoping Review (Preprint)

“…Some researchers presented ethical concerns relating to publicly traded companies inferring or collecting sensitive health information about their users and acting on it, or sharing it, without explicit consent [47,78]. Non-transparent data collection and inference processes currently used by social media platforms were highlighted as a growing area of concern [84]. Hallucinations (the tendency for LLMs to sometimes generate false responses), were identified as having impacts for the safety and reliability of LLM applications [47,50].…”

“…Others noted that integration of LLMs into clinical care may lead to a sense of distancing of the clinician from the individual, potentially fostering feelings of invalidation or insignificance [52] and exacerbating suicidal thoughts or self-harm behaviors. False negatives (when suicidality goes undetected) and false positives (when suicidality is incorrectly flagged as being present) were noted as concerns [83,84] as psychological harm can result (e.g., resulting in missed opportunities to intervene with someone at risk, or in unnecessary mental health evaluations for someone who is not at risk) [84]. Safety was also noted as a concern as conversational AI may advance at a pace that outstrips associated safety measures [48].…”

“…Users of social media sites are often not afforded the opportunity to opt out of such surveillance or analysis, nor are they made aware of the potential adverse consequences associated with having their data used in this manner (e.g., the risk of breaches to data privacy, as has happened with some LLMs) [111]. This contravenes the fundamental human right to privacy, recently underscored by European legislation prohibiting certain data gathering practices related to health information by the algorithms of Facebook and other large social media companies [84,112]. The use of social media and other data to train LLMs requires deep discussion to discern the regional, social, cultural, and temporal factors (among others) that influence ethical decisions around safe use of this data.…”

Detecting Substance Use Disorder Using Social Media Data and the Dark Web: Time- and Knowledge-Aware Study

“…A sparse additive generative model, a topic analysis tool, was used to assess the temporal linguistic changes in tweets with and without evidence of self-harm. Furthermore, they explored temporal linguistic features of tweets with and without suicidal intent signs [ 23 ]. A transformer-based model was also proposed for suicidal ideation detection in social media that takes into consideration the temporal context [ 24 ].…”