DeStein: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion
Research Area: Alignment, Safety, Inference algorithms for LMs
Keywords: detoxification; controllable text generation; language models
TL;DR: We propose a novel approach for detoxifying language models by editing representations in the activation space, surpassing previous state-of-the-art models on detoxification metrics and offering easy scalability to larger models.
Abstract: Despite the remarkable achievements of language models (LMs) across a broad spectrum of tasks, their propensity for generating toxic outputs remains a prevalent concern. Current solutions involving finetuning or auxiliary models usually require extensive computational resources, hindering their practicality in large language models (LLMs). In this paper, we propose DeStein, a novel method that detoxifies LMs by applying representation engineering in activation spaces with lower resource and time costs. Specifically, we derive detoxification vectors from self-induced, universal steering pairs through arithmetic operations in activation spaces. During inference, detoxification is achieved by fusing the detoxification vectors with the original representations in a head-wise manner. Empirical results demonstrate that our method significantly outperforms previous state-of-the-art approaches on various metrics, while also maintaining satisfactory generation quality and diversity. We further validate the practicality and scalability of DeStein with a series of white-box LLMs. Warning: Some example model outputs may contain highly offensive or disturbing text.
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html
Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html
Submission Number: 815
Loading