A Framework of Severity for Harmful Content Online

Abstract

The proliferation of harmful content on online social media platforms has necessitated empirical understandings of experiences of harm online and the development of practices for harm mitigation. Both understandings of harm and approaches to mitigating that harm, often through content moderation, have implicitly embedded frameworks of prioritization-what forms of harm should be researched, how policy on harmful content should be implemented, and how harmful content should be moderated. To aid efforts of better understanding the variety of online harms, how they relate to one another, and how to prioritize harms relevant to research, policy, and practice, we present a theoretical framework of severity for harmful online content. By employing a grounded theory approach, we developed a framework of severity based on interviews and card-sorting activities conducted with 52 participants over the course of ten months. Through our analysis, we identified four Types of Harm (physical, emotional, relational, and financial) and eight Dimensions along which the severity of harm can be understood (perspectives, intent, agency, experience, scale, urgency, vulnerability, sphere). We describe how our framework can be applied to both research and policy settings towards deeper understandings of specific forms of harm (e.g., harassment) and prioritization frameworks when implementing policies encompassing many forms of harm.CCS Concepts: • Human-centered computing → Empirical studies in HCI; HCI theory, concepts and models.

Safe from “harm”: The governance of violence by platforms

“…Although typologizing the kinds of harms that can occur is necessary to moderate it (Banko et al, 2020; Banks, 2010), what platforms often fail to do is meaningfully repair what leads to harmful behavior and content in the first place. Moreover, platforms also attempt to assign point values or quantitatively make decisions on what is or is not harmful content, which also can lead to overlooking or outright allowing certain harms to persist if they do not meet a certain numeric threshold, which perpetuates the notion that certain kinds of harms are prioritized over others (Scheuerman et al, 2021). Despite platforms implementing these policies and continuing to update what constitutes hate speech and harassment and their attempts to moderate this kind of content, inconsistencies in these definitions abound within platforms as well as across them (Pater et al, 2016).…”

Making Online Communities 'Better': A Taxonomy of Community Values on Reddit

“…In cases of extreme intracommunity conflict, such tensions can even lead to fragmentation as some members exit [26] and form new, alternative communities [23,64]. Additional evidence for intracommunity tension can be found in work that examines how members define rulebreaking and how to fairly punish such behavior; this work has found substantial disagreement amongst community members [43,70].…”

“…Even when values are shared by all community members, there are likely to be differences in their perceived importance across members. Some work studying perceptions of harmful behavior already provide evidence for differences in opinions on values within a community [43,70]. In the context of peer production communities such as Wikipedia, some research has already explored some sources of internal disagreement, such as tensions between senior and junior members [34,35,80], however the extent to which these findings generalize to social media platforms such as reddit may be limited.…”

“…Previous work on online communities has largely focused on specific aspects of online communities, mostly harmful ones, such as harassment [11,61], rule-breaking [14,24,60], and fake news [3,8,30,36,74,87]. However, upon deeper inquiry, even these more commonly studied harms are quite complex and difficult to pin down, with substantial 1 disagreement within and across communities regarding the extent to which these ostensibly harmful behaviors should be tolerated [41,43,57,70].…”

From Scalability to Subsidiarity in Addressing Online Harm

“…Online harm occurs in a variety of ways and generally takes two related forms: individually targeted harassment (Vogels, 2021) and harmful content that may be targeted at a group or at no one in particular (Scheuerman et al, 2021). The lack of context sensitivity in the responses to all forms of harm appears to be a widespread pattern among social platforms.…”