Strategy Coopetition Explains the Emergence and Transience of In-Context Learning

Aaditya K Singh, Ted Moskovitz, Sara Dragutinović, Felix Hill, Stephanie C.Y. Chan, Andrew M Saxe

TL;DR: We find and model cooperative and competitive dynamics (termed "coopetition") that explain the emergence and subsequent transience of in-context learning.

Abstract:

In-context learning (ICL) is a powerful ability that emerges in transformer models, enabling them to learn from context without weight updates. Recent work has established emergent ICL as a transient phenomenon that can sometimes disappear after long training times. In this work, we sought a mechanistic understanding of these transient dynamics. Firstly, we find that—after the disappearance of ICL—the asymptotic strategy is a remarkable hybrid between in-weights and in-context learning, which we term “context-constrained in-weights learning” (CIWL). CIWL is in competition with ICL, and eventually replaces it as the dominant strategy of the model (thus leading to ICL transience). However, we also find that the two competing strategies actually share sub-circuits, which gives rise to cooperative dynamics as well. For example, in our setup, ICL is unable to emerge quickly on its own, and can only be enabled through the simultaneous slow development of asymptotic CIWL. CIWL thus both cooperates and competes with ICL, a phenomenon we term “strategy coopetition”. We
propose a minimal mathematical model that reproduces these key dynamics and interactions. Informed by this model, we were able to identify a setup where ICL is truly emergent and persistent.

Lay Summary: In-context learning (ICL) is a key ability of modern large language models, allowing them to learn from context and adapt to new situations without being explicitly trained for them. Prior work has found that, in many cases of overtraining, classical in-context learning is actually transient, meaning it appears but then fades away. We reproduce this finding in a smaller scale setup, thus enabling mechanistic investigation. Through these investigations, we find that the asymptotic strategy (which emerges after ICL transience) is context-constrained in-weight learning (CIWL) -- namely, the model relies on associations learned in the past, but constrains the recall based on the current context. Competition for capacity leads CIWL to "crowd out" ICL, but surprisingly, cooperation between the two strategies is what leads to ICL emergence in the first place. We capture these "coopetition" dynamics in a toy model, which motivates an empirical setting where ICL is emergent and persistent. Overall, this work deepens our mechanistic understanding of how different strategies may tradeoff through during the course of transformer training.

Link To Code: github.com/aadityasingh/icl-dynamics

Primary Area: Deep Learning->Attention Mechanisms

Keywords: Mechanistic interpretability, transformers, in-context learning, transience, dynamics, Machine Learning, strategy, cooperation, competition, coopetition

Submission Number: 5794