Recast

Abstract

With the widespread use of toxic language online, platforms are increasingly using automated systems that leverage advances in natural language processing to automatically flag and remove toxic comments. However, most automated systems, when detecting and moderating toxic language, do not provide feedback to their users, let alone provide an avenue of recourse for these users to make actionable changes. We present our work, RECAST, an interactive, open-sourced web tool for visualizing these models' toxic predictions, while providing alternative suggestions for flagged toxic language. Our work also provides users with a new path of recourse when using these automated moderation tools. RECAST highlights text responsible for classifying toxicity, and allows users to interactively substitute potentially toxic phrases with neutral alternatives. We examined the effect of RECAST via two large-scale user evaluations, and found that RECAST was highly effective at helping users reduce toxicity as detected through the model. Users also gained a stronger understanding of the underlying toxicity criterion used by black-box models, enabling transparency and recourse. In addition, we found that when users focus on optimizing language for these models instead of their own judgement (which is the implied incentive and goal of deploying automated models), these models cease to be effective classifiers of toxicity compared to human annotations. This opens a discussion for how toxicity detection models work and should work, and their effect on the future of online discourse.

Farsight: Fostering Responsible AI Awareness During AI Application Prototyping

“…Using in-context warnings to improve users' safety awareness and encourage users to take protection measures can be considered a form of "digital nudging" [26,160]. More recently, researchers have also adapted in-context security warnings to nudge social media users to recognize and avoid online disinformation [85,163] and reflect before posting potentially harmful content [88,169,200]. Beyond platform-initiated integration of warnings, end-users also voluntarily seek in-context alert interfaces for productivity improvement.…”

A Systematic Review of Toxicity in Large Language Models: Definitions, Datasets, Detectors, Detoxification Methods and Challenges

“…Referenced LSTM [37,52,55,60,64,75,79,80,82,86,96,97,107,116,122,128,131,134,167,203,216,217,219,220] Dense Neural Networks [216] CNN + LSTM [38,55,129] LSTM + CNN [38,75,79] M-BERT [236] [4, 42, 195, 200] RoBERTa [243] [42, 45, 79, 90, 107, 114, 123, 136-138, 147, 155, 157-161, 168-171, 176, 179, 203, 209, 212, 216] DistillBERT [244] [42, 60, 90, 106, 111, 117, 146, 147, 154, 167, 175, 198, 202, 213] CNN [37, 43, 47, 53, 55, 60, 64, 73, 80, 83, 96-98, 112, 119, 122, 144, 146, 210, 216, 221] HateBERT [245] [9, 146, 165, 207] ToxDectRoBERTa [209] [9, 207] ALBERT [246] [46, 90, 146, 147, 155, 157, 176, 202] ALBERT+BILSTM+CRF [162] CNN-CAPSULE [98] LSTM-CAPSULE [98] CapsNet [247] [99]…”

Measuring Toxicity Toward Women in Game-Based Communities

“…Our stated position for this score is that it be used either as part of a discovery process, for descriptive reporting or purposes, or for self-assessment. The RECAST tool created by Wright et al (2021) shows how users would willingly substitute nontoxic language for toxic language when given the opportunity with objectionable posts they had made. Recalling the adage of mice and mousetraps, however, other users instead learned how to, more cleverly, elude the toxicity detection system.…”

“…The same method that, for our research, can measure toxicity for a forum could just as easily measure toxicity for an individual. If this information could be candidly reported to individuals, then those individuals would have an opportunity to reflect on their contributions to toxicity and alter their patterns of interaction (Wright et al, 2021). Whether individuals would alter their behavior is beyond the purview of this current research, but to be able to provide a reliable measurement of toxicity that could be reported is an intermediate step in achieving such future aims.…”