The Psychological Well-Being of Content Moderators

Steiger, Miriah; Bharucha, Timir J.; Venkatagiri, Sukrit; Riedl, Martin; Lease, Matthew

doi:10.1145/3411764.3445092

Cited by 93 publications

(54 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In line with recent work by Steiger et al [89] to provide necessary programmatic and technological interventions to offer resources and assistance for mental health, we provide several design interventions that have the potential to protect content moderators or at least mitigate the position's ill effects. Due to the results of positive emotional stimulation being a net negative in the present work, we propose interventions that take the opposite approach and encourage alternative break-time strategies.…”

Section: Practical and Design Implicationsmentioning

confidence: 90%

“…However, researchers have begun to take interest in the wellbeing of content moderators [61,75,89] and have examined the effectiveness of current methods to ease the burden on content moderators (e.g., [13,15,79,80]), referring to both moderators in volunteer and commercial content moderation, as they both experience exposure to harmful content and suffer from emotional labors [89]. Researchers have tried to reduce the burden on content moderators by altering the offensive content to reduce exposure, often by blurring image and video content [20,22], and by either supporting or replacing human moderators with artificial intelligence systems, to varying degrees of success [76].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Awe Versus Aww: The Effectiveness of Two Kinds of Positive Emotional Stimulation on Stress Reduction for Online Content Moderators

Cook¹,

Cai²,

Wohn³

2022

Preprint

View full text Add to dashboard Cite

When people have the freedom to create and post content on the internet, particularly anonymously, they do not always respect the rules and regulations of the websites on which they post, leaving other unsuspecting users vulnerable to sexism, racism, threats, and other unacceptable content in their daily cyberspace diet. However, content moderators witness the worst of humanity on a daily basis in place of the average netizen. This takes its toll on moderators, causing stress, fatigue, and emotional distress akin to the symptomology of post-traumatic stress disorder (PTSD). The goal of the present study was to explore whether adding positive stimuli to breaktimes-images of baby animals or beautiful, aweinspiring landscapes-could help reduce the negative side-effects of being a content moderator. To test this, we had over 300 experienced content moderators read and decide whether 200 fake text-based social media posts were acceptable or not for public consumption. Although we set out to test positive emotional stimulation, however, we actually found that it is the cumulative nature of the negative emotions that likely negates most of the effects of the intervention: the longer the person had practiced content moderation, the stronger their negative experience. Connections to compassion fatigue and how best to spend work breaks as a content moderator are discussed.CCS Concepts: • Human-centered computing → Empirical studies in collaborative and social computing; Empirical studies in HCI .

show abstract

Section: Practical and Design Implicationsmentioning

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Awe Versus Aww: The Effectiveness of Two Kinds of Positive Emotional Stimulation on Stress Reduction for Online Content Moderators

Cook¹,

Cai²,

Wohn³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The Linguistic Data Consortium (LDC), for example, arranged stress-relieving activities for annotators of broadcast news data, following reports of "negative psychological impact" such as intense irritation, overwhelmed feelings, and task-related nightmares (Strassel et al, 2000). Although literature on the emotional toll on crowdworkers is still scant (Huws, 2015), there is growing literature on the psychological cost of work done by commercial content moderators (Steiger et al, 2021). Crowdworkers deserve similar consideration: while NLP tasks can be as benign as the POS tagging of a children's poem, they may also involve exposure to disturbing textual or visual content.…”

Section: Inducing Psychological Harmsmentioning

confidence: 99%

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

Shmueli¹,

Fell²,

Ray³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

The use of crowdworkers in NLP research is growing rapidly, in tandem with the exponential increase in research production in machine learning and AI. Ethical discussion regarding the use of crowdworkers within the NLP research community is typically confined in scope to issues related to labor conditions such as fair pay. We draw attention to the lack of ethical considerations related to the various tasks performed by workers, including labeling, evaluation, and production. We find that the Final Rule, the common ethical framework used by researchers, did not anticipate the use of online crowdsourcing platforms for data collection, resulting in gaps between the spirit and practice of human-subjects ethics in NLP research. We enumerate common scenarios where crowdworkers performing NLP tasks are at risk of harm. We thus recommend that researchers evaluate these risks by considering the three ethical principles set up by the Belmont Report. We also clarify some common misconceptions regarding the Institutional Review Board (IRB) application. We hope this paper will serve to reopen the discussion within our community regarding the ethical use of crowdworkers.

show abstract

“…Mods also suffer from emotional tolls like lack of appreciation from the community administrator [79] and have to handle the emotional labor [22]. They might even experience impairment of psychological well-being [70] due to facing harmful content and doing the dirty work for a long time [58]. Though a lot of research explores how mods handle violations and their impact, there is a lack of understanding of how mods perceive violators/bad actors.…”

Section: Content Moderationmentioning

confidence: 99%

“…The need of understanding evidence collection in community moderation and the lack of framework in user profiling literature to understand violators indicate the potential of a new lens to build a connection between community moderation and profiling research. Additionally, much research also introduces various types of justice (e.g., social justice, retributive justice, restorative justice) from the criminal justice system to explain online harassment and moderation, and the justice-seeking process [2,21,60,61,70]. The inherent role of criminal profiling in the criminal justice system and its components (evidence collection and analysis, and the deduction of offender's characteristics) in the definition suggests that criminal profiling may serve as a good lens to understand mods' profiling process when they face potential violators at a conceptual level.…”

Section: Applying Criminal Profiling To Community Moderation and In Live Streamingmentioning

confidence: 99%

After Violation But Before Sanction: Understanding Volunteer Moderators' Profiling Processes Toward Violators in Live Streaming Communities

Cai

Wohn

2021

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Content moderation is an essential part of online community health and governance. While much of extant research is centered on what happens to the content, moderation also involves the management of violators. This study focuses on how moderators (mods) make decisions about their actions after the violation takes place but before the sanction by examining how they "profile" the violators. Through observations and interviews with volunteer mods on Twitch, we found that mods engage in a complex process of collaborative evidence collection and profile violators into different categories to decide the type and extent of punishment. Mods consider violators' characteristics as well as behavioral history and violation context before taking moderation action. The main purpose of the profiling was to avoid excessive punishment and aim to integrate violators more into the community. We discuss the contributions of profiling to moderation practice and suggest design mechanisms to facilitate mods' profiling processes.CCS Concepts: • Human-centered computing → Empirical studies in HCI.

show abstract

The Psychological Well-Being of Content Moderators

Cited by 93 publications

References 68 publications

Awe Versus Aww: The Effectiveness of Two Kinds of Positive Emotional Stimulation on Stress Reduction for Online Content Moderators

Awe Versus Aww: The Effectiveness of Two Kinds of Positive Emotional Stimulation on Stress Reduction for Online Content Moderators

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

After Violation But Before Sanction: Understanding Volunteer Moderators' Profiling Processes Toward Violators in Live Streaming Communities

Contact Info

Product

Resources

About