SoK: Hate, Harassment, and the Changing Landscape of Online Abuse

Abstract

In this paper, we explore the feasibility of leveraging large language models (LLMs) to automate or otherwise assist human raters with identifying harmful content including hate speech, harassment, violent extremism, and election misinformation. Using a dataset of 50,000 comments, we demonstrate that LLMs can achieve 90% accuracy when compared to human verdicts. We explore how to best leverage these capabilities, proposing five design patterns that integrate LLMs with human rating, such as pre-filtering non-violative content, detecting potential errors in human rating, or surfacing critical context to support human rating. We outline how to support all of these design patterns using a single, optimized prompt. Beyond these synthetic experiments, we share how piloting our proposed techniques in a real-world review queue yielded a 41.5% improvement in optimizing available human rater capacity, and a 9-11% increase (absolute) in precision and recall for detecting violative content.

Designing Toxic Content Classification for a Diversity of Perspectives

“…We use the term toxic content as an umbrella for identitybased attacks such as anti-Semitism or racism posted publicly to social media [19,37,55], bullying in online gaming or replies to posts [35,50], trolling [8], threats of violence, sexual harassment, and more [47,52]. These attacks represent just a subset of abuse stemming from hate and harassment, a much broader threat that encompasses any activity where an attacker attempts to inflict emotional harm on a target (e.g., stalking, doxxing, sextortion, and intimate partner violence) [9,52]. Unlike spam, phishing, or related abuse classification problems that can rely on expert raters, toxic content is an inherently subjective problem as we show in our work.…”

“…Online hate and harassment is a pernicious threat facing 48% of Internet users [52]. In response to this growing challenge, online platforms have developed automated tools to take action against toxic content (e.g., hate speech, threats, identity attacks).…”

“…These "gray areas" stem from the fact that users may disagree about what constitutes toxic content online based on their lived experiences, cultural perspective, political views towards free speech, or access to appropriate context [24,50]. While prior research has demonstrated that certain groups are more at-risk of experiencing online hate and harassment [45,52], no study has investigated how users from diverse backgrounds interpret online toxicity or how their views on what content they would like to see online differ. Understanding these nuanced differences is an important first step to designing harassment defenses for diverse Internet users.…”

Towards Stalkerware Detection with Precise Warnings

“…Technology-enabled IPV is a troubling phenomenon that fits into the broader ecosystem of online hate, harassment, and abuse [49]. Its manifestations include many forms of harassment [6,23,39], character assassination through faked revenge porn [43], impersonation attacks that damage the targeted individual's relationships [25,40], and above all, spying on an intimate partner through stalkerware and other means, such as knowledge of the survivor's account credentials [24,25].…”

“…In the U.S., 15.8% of women and 5.3% of men reported being subjected to stalking violence "in which they felt very fearful or believed that they or someone close to them would be harmed or killed" [8]. IPV survivors have shed light on the many ways in which technology plays a role in inter-personal attacks [8,20,40,43,49], of which tech-enabled stalking and spying by current or former romantic partners are especially common and pernicious [24,34]. In a recent survey, 10% of the U.S. adult respondents admitted to using a mobile phone app to spy on an intimate partner [50].…”