In-place Implicit In-context Learning by Neuron Amplification

ACL ARR 2025 February Submission6860 Authors

Abstract: Large language models (LLMs) have achieved remarkable performance through in-context learning (ICL) with demonstrations, yet these methods incur significant GPU memory and computational costs. Therefore, we consider a cost-efficient approach to implement implicit ICL such that demonstrations do not occupy space in the context. In this paper, we propose an in-place method for implicit ICL by identify- ing and amplifying specific neurons within the feed-forward networks of LLMs. The proposed method transfers few-shot learning capabilities to zero-shot settings through neuron perturba- tion. Despite the model taking zero-shot inputs, our method leads to performance approaching few-shot learning, while requiring no additional computation or memory costs. Experimental re- sults across instruction-following and problem- solving tasks demonstrate that our approach enables implicit ICL.

Paper Type: Short

Research Area: Machine Learning for NLP

Research Area Keywords: few-shot learning, in-context learning

Contribution Types: Model analysis & interpretability, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 6860