Enhancing Cross-task Transfer of Large Language Models via Fourier Activation Steering

Xinyu Tang, Zhihao Lv, Yu-Liang Zhan, Xiaoxue Cheng, Junyi Li, Xin Zhao, zujie wen, Zhiqiang Zhang, JUN ZHOU

Keywords: Cross-task transfer, Fourier activation steering

TL;DR: We observe that cross-task transfer can be achieved via latent space steering and propose Fourier-based activation steering framework for cross-task transfer.

Abstract:

Large Language Models (LLMs) have shown impressive abilities in leveraging pretrained knowledge through prompting, but they often struggle with unseen tasks, especially in data-scarce scenarios.
While cross-task in-context learning provides a direct solution for knowledge transfer without fine-tuning, it still faces limitations in terms of robustness, scalability, and efficiency.
In this paper, we investigate whether cross-task transfer can be achieved via latent space steering.
Through analysis of activation patterns under both zero-shot and few-shot prompts, we have three observations:
(1) the activation differences between few-shot and zero-shot prompts exhibit a nearly parallel structure in low-dimensional space;
(2) these difference vectors correlate strongly with task similarity;
(3) Fourier analysis reveals that low-frequency components encode task-agnostic, information-enhanced features, while high-frequency components capture task-specific details.
Motivated by these findings, we propose FAST, a Fourier-based Activation Steering Transfer framework.
It first selects influential and diverse samples from high-resource tasks, then injects information-enhanced low-frequency components along with task-similarity weighted high-frequency components during inference.
Extensive experiments in both cross-domain and cross-lingual transfer settings show that our method consistently outperforms existing methods.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 10190