In-Context Learning Without Copying

ICLR 2026 Conference Submission20514 Authors

Keywords: In-Context Learning, Induction Heads, Large Language Models, Training Dynamics, Mechanistic Interpretability

TL;DR: Hapax suppresses copying, weakening induction heads but leaving abstractive ICL intact, and often better than vanilla, showing copying is not essential for learning abstractive ICL.

Abstract: Induction heads are attention heads that perform inductive copying by matching patterns from earlier context and copying their continuations verbatim. As models develop induction heads, they often experience a sharp drop in training loss, a phenomenon cited as evidence that induction heads may serve as a prerequisite for more complex in-context learning (ICL) capabilities. In this work, we ask whether transformers can still acquire ICL capabilities when inductive copying is suppressed. We propose Hapax, a setting where we omit the loss contribution of any token that can be correctly predicted by induction heads. Despite a significant reduction in inductive copying, performance on abstractive ICL tasks (i.e., tasks where the answer is not contained in the input context) remains comparable and surpasses the vanilla model on 12 of 18 tasks even though 31.7\% of tokens are omitted from the loss. Furthermore, our model achieves lower loss values on token positions that cannot be predicted correctly by induction heads. Mechanistic analysis further shows that models trained with Hapax develop fewer and weaker induction heads but still preserve ICL capabilities. Taken together, our findings indicate that inductive copying is not essential for learning abstractive ICL mechanisms.

Primary Area: interpretability and explainable AI

Submission Number: 20514