When Bad Data Leads to Good Models

Kenneth Li, Yida Chen, Fernanda Viégas, Martin Wattenberg

TL;DR: A little bit of toxic data in pretraining can act as a catalyst for more alignable language models.

Abstract: In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

Lay Summary: This paper challenges the usual idea that cleaner training data always leads to better language models. It shows that training a model with some toxic data can actually make it easier to reduce harmful outputs later on. Through experiments, the authors find that toxic content helps the model learn clearer representations of toxicity, which in turn makes it easier to control. When post-training methods are used to reduce toxicity, models trained with some toxic data perform better overall---maintaining their abilities while being safer.

Primary Area: Deep Learning->Large Language Models

Keywords: Large Language Models, Toxicity

Submission Number: 14342