Iterative Vectors: Boost In-Context Learning within Activations
Keywords: Large Language Model, In-context Learning
TL;DR: Iterative Vectors (IVs) enhance in-context learning by steering activations in language models through forward inference passes, leading to significant performance improvements without backpropagation.
Abstract: In-context learning (ICL) has emerged as a standard paradigm for utilizing language models. Although ICL is convenient due to the absence of backpropagation, selecting and processing appropriate demonstration examples can be difficult and time-consuming, particularly when the number of examples is large. We propose to explore the potential of activation space through Iterative Vectors (IVs), a technique designed to enhance in-context performance and necessitating only forward inference passes. IVs are employed by first extracting and iteratively steering activations within a language model, then applying them during inference with minimal computational and memory overhead. We evaluate IVs across numerous tasks using four popular models and observe significant improvements. Our findings suggest that activation steering can serve as a promising direction for in-context learning, thereby opening new avenues for future research.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 2279
Loading