Causal-Steer: Disentangled Continuous Style Control without Parallel Corpora

Qingsong Wang, Chang Yao, Jingyuan Chen

Keywords: Controllable Generation, Activation Steering, Style Control, Large Language Models

TL;DR: We treat LoRA as a causal probe to extract a pure "style vector" from non-parallel corpora. This vector allows for continuous, bidirectional control over text style at inference time with almost no computational cost.

Abstract: Controlling stylistic attributes of Large Language Models (LLMs), such as formality or conceptual complexity, is crucial for effective human-AI interaction. However, current methods often suffer from discreteness, reliance on expensive parallel corpora, and instability, limiting their practical utility. This paper introduces a novel framework for robust activation steering that eliminates the need for parallel corpora, enabling continuous, fine-grained, and linear control over LLM outputs. Our key insight is to reframe Low-Rank Adaptation (LoRA) as a causal intervention tool. By contrasting activations on identical inputs with and without a LoRA perturbation trained via a contrastive objective, we separate the influence of content. To enhance reliability, we introduce a robust aggregation pipeline that uses Principal Component Analysis (PCA) for denoising and the geometric median for centrality estimation, yielding a stable and disentangled style vector. At inference, this vector allows for precise bidirectional control via activation steering with negligible computational overhead. We demonstrate state-of-the-art performance on controlling conceptual complexity, text detoxification, and formality control. Our method not only provides superior control but also generalizes across different models and tasks, and enables simultaneous multi-attribute control.

Primary Area: generative models

Submission Number: 8612