Solve the Loop: Attractor Models for Language and Reasoning

Jacob Fein-Ashley, Paria Rashidinejad

Keywords: Reasoning, Large Language Models, New Architectures

TL;DR: We develop Attractor Models, which turn recurrent computation from an unrolled trajectory into a solved latent state, yielding a single scalable architecture that improves both large-scale language modeling and hard reasoning tasks with tiny models.

Abstract: Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce \emph{Attractor Models}, in which a backbone module first propose output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, this yields a single mechanism that outperforms existing models and scales across two regimes, large-scale language modeling and reasoning with tiny models. In language modeling, Attractor Models outperform standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6\% and downstream accuracy by up to 19.7\% while reducing training cost. On challenging reasoning tasks, we show that our model with only 27M parameters and around 1000 examples achieves 91.4\% accuracy on Sudoku-Extreme and 93.1\% on Maze-Hard, scaling favorably where frontier models fail completely and specialized recursive reasoners collapse at larger sizes. Lastly, we show Attractor Models enjoy a novel phenomenon, which we call \emph{equilibrium internalization}: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference-time with little degradation.

Submission Number: 208