Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations

ICLR 2026 Conference Submission13579 Authors

Keywords: transformer, in-context learning, task vector

TL;DR: We explain how task vectors emerge and function in in-context learning, and point out their limitations.

Abstract: Task vector is a compelling mechanism for accelerating inference in in-context learning (ICL) by distilling task-specific information into a single, reusable representation. Despite their empirical success, the underlying principles governing their emergence and functionality remain unclear. This work proposes the *Task Vectors as Representative Demonstrations* conjecture, positing that task vectors encode single in-context demonstrations distilled from the original ones. We provide both theoretical and empirical support for this conjecture. First, we show that task vectors naturally emerge in linear transformers trained on triplet-formatted prompts through loss landscape analysis. Next, we predict the failure of task vectors in representing high-rank mappings and confirm this on practical LLMs. Our findings are further validated through saliency analyses and parameter visualization, suggesting an enhancement of task vectors by injecting multiple ones into few-shot prompts. Together, our results advance the understanding of task vectors and shed light on the mechanisms underlying ICL in transformer-based models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 13579