Coherence–Diffusion Dynamics: A Continuous-Semantic Interpretation of Transformer Language Models

TMLR Paper6852 Authors

Abstract: Despite strong empirical performance, the principles governing how large language models organize and stabilize semantic meaning during inference remain poorly understood. We focus on three recurrent empirical patterns that are not readily explained by existing approaches: (i) paradoxically higher attention concentration at incorrect token positions, (ii) a depth-dependent sign reversal in the relationship between predictive uncertainty and instability for incorrect tokens, and (iii) convergence of multiple signals derived from distinct empirical sources to a narrow depth range. We introduce the Coherence--Diffusion Dynamics (CDD) framework, which interprets Transformer inference as the evolution of a latent semantic trajectory under the joint influence of coherence-restoring drift and stochastic variability. Rather than positing new phenomena, CDD provides a unifying, phenomenological account that organizes these patterns into a regime-structured view of inference-time dynamics. Within this framework, we formulate four falsifiable structural constraints (C1--C4) on observable proxies of instability, diffusion, and coherence. Constraints C1--C3 formalize the observed empirical regularities as joint consistency requirements that frameworks lacking regime-level structure would not be expected to account for all three simultaneously within a single mechanism. C4 constitutes a derived, testable implication arising from the non-linear interaction between drift and diffusion. We evaluate these constraints on GPT-2 Large using inference-time measurements and examine cross-architecture consistency on Pythia-1.4B. On GPT-2 Large, all four constraints (C1--C4) are satisfied. On Pythia-1.4B, C1 and C2 replicate, while C3 and C4 are partially supported due to a broadened convergence window and entropy-induced sparsity in deeper layers. These findings provide a falsifiable, empirically grounded account of semantic trajectory dynamics and demonstrate that inference-time behavior exhibits regime-level structure that cannot be explained by independent or single-proxy analyses.

Submission Type: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

The manuscript has been substantially revised in response to reviewer feedback. The key changes are as follows.

**Conceptual framing.** The original presentation as a unifying interpretation has been reformulated into a constraint-based, falsifiable framework (Section 3.3). We introduce four explicit structural constraints (C1–C4), each with clearly defined falsification conditions, shifting the contribution from descriptive compatibility to testable structure.

**Empirical redesign.** The original training-based experiment has been replaced with a direct inference-time trajectory analysis (Section 5), using teacher-forced forward passes to measure layer-wise evolution of surprisal, entropy, and coherence proxies. This resolves the mismatch between the theory's scope (inference dynamics) and the original evaluation. Additional analyses include coherence–diffusion coupling (sign reversal of corr(σ, ΔΨ) for incorrect positions), layer-wise pruning sensitivity across all transformer blocks, and three-signal convergence to a common critical depth range (C3).

**Proxy grounding.** All proxy definitions have been clarified and grounded in explicit measurement procedures. We now distinguish clearly between latent conceptual quantities (α, σ, Ψ) and their operational proxies, and provide architectural and empirical justification for each, including citations to prior work on attention concentration, residual alignment, and entropy-based uncertainty.

**Extended scope.** Core results are replicated on Pythia-1.4B, providing cross-architecture evidence for the qualitative regime structure identified in GPT-2 Large. Dynamic sparsity experiments have been extended to a full layer-wise analysis (Experiment III, Part B).
Presentation. The manuscript has been substantially reorganized to reduce redundancy and improve clarity. Repetitive sections have been removed, the experimental structure has been aligned with the falsifiable constraints (C1–C4), and the related work section has been expanded to cover continuous-dynamics Transformer literature and eigenspectral methods.

Assigned Action Editor: ~Surbhi_Goel1

Submission Number: 6852