Towards Understanding Steering Strength

Magamed Taimeskhanov, Samuel Vaiter, Damien Garreau

TL;DR: We analyze theoretically how steering strength affects next-token probabilities, concept probability, and cross-entropy.

Abstract: A popular approach to post-training control of large language models (LLMs) is the steering of intermediate latent representations. Namely, identify a well-chosen direction depending on the task at hand and perturbs representations along this direction at inference time. While many propositions exist to pick this direction, considerably less is understood about how to choose the magnitude of the move, whereas its importance is clear: too little and the intended behavior does not emerge, too much and the model's performance degrades beyond repair. In this work, we propose the first theoretical analysis of steering strength. We characterize its effect on next token probability, presence of a concept, and cross-entropy, deriving precise qualitative laws governing these quantities. Our analysis reveals surprising behaviors, including non-monotonic effects of steering strength. We validate our theoretical predictions empirically on eleven language models, ranging from a small GPT architecture to modern models.

Lay Summary: Large language models can learn unwanted behaviors from their training data, and practitioners often try to control them without retraining the whole model. One lightweight method is to find a ``direction'' inside the model that is associated with a behavior, such as safe or unsafe answers, and nudge the model along that direction while it is generating text. But this raises a practical question: how hard should we push? Too small a nudge may do nothing, while too large a nudge can make the model's answers worse or repetitive. We give the first mathematical analysis of this steering strength for a widely used steering method, showing how it changes the model's next-word choices, the presence of the desired concept, and overall prediction quality. We also test our theoretical predictions on eleven language models of different sizes and find the same qualitative patterns. This understanding can help researchers choose steering strength in a more principled way when making language models safer.

Link To Code: https://github.com/MagamedT/steering

Primary Area: Theory->Deep Learning

Keywords: Steering, Large Language Models, Theory, Alignment

Submission Number: 26991