Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

Abstract

Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pretraining corpus. We demonstrate that the selfgeneration method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1 3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3× larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.

Contemporary Approaches in Evolving Language Models

“…Perplexity-a measure of how well a language model predicts the next word in a sequence [82][83][84][85][86][87][88]. A lower perplexity value indicates that the language model is better at predicting the subsequent word (9).…”