Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineering

Joris Postmus, Steven Abreu

Keywords: activation engineering, mechanistic interventions, model steering, large language models, activation addition, function vectors

TL;DR: We propose a new mechanism and formalism for activation engineering that improves addition-based approaches

Abstract: Large language models have transformed AI, yet reliably controlling their outputs remains a challenge. This paper explores activation engineering, where outputs of pre-trained LLMs are controlled by manipulating their activations at inference time. Unlike traditional methods using a single steering vector, we introduce conceptors - mathematical constructs that represent sets of activation vectors as ellipsoidal regions. Conceptors act as soft projection matrices and offer more precise control over complex activation patterns. Our experiments demonstrate that conceptors outperform traditional methods across multiple steering tasks. We further use Boolean operations on conceptors for combined steering goals that empirically outperform additively combining steering vectors on a set of tasks. These results highlight conceptors as a promising tool for more effective steering of LLMs. Our code is available on [github.com/jorispos/conceptorsteering](https://github.com/jorispos/conceptorsteering).

Email Of Author Nominated As Reviewer: [e-mail]

Submission Number: 25