Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Context

ICLR 2026 Conference Submission21902 Authors

Keywords: Interpretability, Entity Binding, Causal Abstraction, Mechanistic Interpretability

TL;DR: Entity binding in LMs is crucial for reasoning. Prior work established a positional mechanism underlying binding, yet it breaks down in complex settings. We find two additional mechanisms, lexical and reflexive, that drive model behavior.

Abstract:

A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval.
For example, an LM might represent *Ann loves pie* by binding *Ann* to *pie*, allowing it to later retrieve *Ann* when asked *Who loves pie?*
Prior research on short lists of bound entities found strong evidence that LMs implement such retrieval
via a **positional mechanism**, where *Ann* is retrieved based on its position in context.
In this work, we find that this mechanism generalizes poorly to more complex settings; as the number of bound entities in context increases, the positional mechanism becomes noisy and unreliable in middle positions.
To compensate for this, we find that LMs supplement the positional mechanism with a **lexical mechanism** (retrieving *Ann* using its bound counterpart *pie*) and a **reflexive mechanism** (retrieving *Ann* through a direct pointer).
Through extensive experiments on nine models and ten binding tasks, we uncover a consistent pattern in how LMs mix these mechanisms to drive model behavior.
We leverage these insights to develop a causal model combining all three mechanisms that estimates next token distributions with 95\% agreement.
Finally, we show that our model generalizes to substantially longer inputs of open-ended text interleaved with entity groups, further demonstrating the robustness of our findings in more natural settings.
Overall, our study establishes a more complete picture of how LMs bind and retrieve entities in-context.

Primary Area: interpretability and explainable AI

Submission Number: 21902