The Progress Helix Tracks Reasoning Depth in Language Models

Andrew Rufail, Andy Huang, Zarreen Reza, Henry Lue, Laurentia Liennart

Keywords: Feature Geometry, Circuit Analysis, Attribution Graphs, Applications of interpretability

TL;DR: LLMs contain progress helices that track and control the output reasoning depth in chain of thought.

Abstract: Large Language Models (LLMs) execute long chains of thought (CoT), but the mechanism by which they maintain a global sense of position within a reasoning trajectory has not been well characterized. To this point, we identify the Progress Helix: an emergent periodic trajectory in a low-dimensional latent subspace that completes exactly one revolution over a generated reasoning chain. Applying PCA and spectral analysis to activations from Llama-3, Gemma-2, and Qwen on both a controlled recursive arithmetic task and GSM8K, we recover a dominant Fourier component at fundamental frequency $k{=}1$ whose magnitude grows monotonically across layers, forming an elliptical cone in activation space. We causally validate the helix's functional role through three interventions. Substituting the natural manifold with a constant-velocity synthetic helix induces logical collapse, indicating polysemanticity with numeric content. Ablating the attention heads most responsible for the $k{=}1$ signal significantly elongates generations. Most strikingly, re-projecting activations onto a helix scaled by speed factor $\gamma$ allows targeted control over output length, with reasoning accuracy peaking at $\gamma{=}0.85$. We thus offer a spectral and topological characterization of reasoning length as a controllable helical degree of freedom in the residual stream.

Submission Number: 454