Distributional Readout: A Memorization Regime in Autoregressive Generative Models

Anany Kotawala

Keywords: memorization, autoregressive models, distributional readout, compositional reasoning, contamination, large language models, generative models

TL;DR: Memorization in autoregressive LLMs is distributional readout: peaked sample-able marginals at each cell, no relational structure across cells. Cross-prompt rank fails; within-prompt rank works.

Abstract: What memorization regime governs frontier autoregressive models that reproduce public numeric series? We argue from the next-token training objective that the regime is distributional readout: the loss induces a peaked conditional marginal at each highly-duplicated value-token but no relational structure across the conditioning set. This predicts rank–value decoupling: high-fidelity sample-based recall does not imply rank access on the same cells. We validate the regime two ways. Across eleven frontier LLMs from five providers, with controls for instruction-following, parser selection, position bias, and within-context capability, the dissociation holds; surfacing both values in one prompt restores >90% rank accuracy, locating the failure at cross-prompt elicitation rather than the marginal itself. A controlled LoRA fine-tuning experiment on an open 1.5B causal LM (Qwen-2.5-1.5B) trained on a synthetic date-indexed series recovers the regime end-to-end, demonstrating training is sufficient. Memorization in current frontier LMs is sample-from-able but not jointly queryable. Code: https://anonymous.4open.science/r/factor-leak-7D4E

Submission Number: 203