Text Detoxification using Large Pre-trained Neural Models

Abstract

We present two novel unsupervised methods for eliminating toxicity in text. Our first method combines two recent ideas: (1) guidance of the generation process with small styleconditional language models and (2) use of paraphrasing models to perform style transfer. We use a well-performing paraphraser guided by style-trained language models to keep the text content and remove toxicity. Our second method uses BERT to replace toxic words with their non-offensive synonyms. We make the method more flexible by enabling BERT to replace mask tokens with a variable number of words. Finally, we present the first largescale comparative study of style transfer models on the task of toxicity removal. We compare our models with a number of methods for style transfer. The models are evaluated in a reference-free way using a combination of unsupervised style transfer metrics. Both methods we suggest yield new SOTA results.

Text simplification as a controlled text style transfer task

“…In it a language model performs text generation guided by another language model conditioned for the specific topic or style or topic. More precisely, in our work, we adopt the extension of this method presented in (Dale et al, 2021), where the authors enable the model not only to generate but to paraphrase the input text. Below, a brief description of the method is given.…”

“…GeDi (Krause et al, 2020) uses a small external language model classifier (or simply GeDi-classifier) to guide the generation of the main language model, re-weighting next token probabilities and, thus, increasing the probabilities of words in the given style. ParaGeDi (Dale et al, 2021) adopts this idea to the paraphrasing task by applying the GeDi approach in combination not with the standard language model but with the paraphraser fine-tuned to rephrase the original text preserving its original meaning.…”

“…For this purpose, we use methods of controllable text generation. Namely, the GeDi algorithm proposed in (Krause et al, 2020) and further developed in (Dale et al, 2021). Following their methodology we use a paraphrase model (the main model) guided by another language model conditioned for the "simple" style (or GeDi-classifier).…”

ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfer

“…Additionally, STRAP has been shown to require large amounts of stylespecific training data (Patel, Andrews, and Callison-Burch 2022). Prior work has explored applying controllable text generation techniques to style transfer (Dale et al 2021;Kumar et al 2021;Mireshghallah, Goyal, and Berg-Kirkpatrick 2022). Our approach is most similar in spirit to Mireshghallah, Goyal, and Berg-Kirkpatrick (2022).…”