Spectral Attention Steering for Prompt Highlighting

ICLR 2026 Conference Submission7673 Authors

Keywords: Spectral learning, Attention steering, Large language models

TL;DR: We propose SEKA, a training-free method that steers attention by editing key embeddings pre-computation. It achieves better performance with negligible overhead.

Abstract: Steering a large language model's attention towards user-specified highlighted text is a critical capability. Existing prompt highlighting methods are incompatible with modern efficient attention mechanisms like Flash Attention due to their reliance on post-hoc matrix editing. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA learns universal relevance subspaces offline via spectral decomposition. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, ensuring full compatibility with optimised attention.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 7673