Architecture, Not Scale: Circuit Localization in Large Language Models

Sohan Venkatesh

Keywords: Circuit Analysis, Attribution Graphs, Benchmarking Interpretability

TL;DR: GQA models have far more concentrated and stable circuits than MHA models, suggesting architecture matters more than scale for mechanistic interpretability.

Abstract: Mechanistic interpretability assumes that circuit analysis becomes harder as models scale. We challenge this assumption by showing that the attention architecture matters more than parameter count. Studying three circuit types across Pythia and Qwen2.5, we find that grouped query attention produces circuits that are far more concentrated and mechanistically stable than standard multi-head attention at comparable scales. The same concentration pattern holds across indirect object identification, induction heads, and factual recall. Within a single architecture family (Qwen2.5), factual recall circuits undergo a discrete phase transition above a critical scale, collapsing to a single bottleneck rather than degrading gradually. These findings suggest that some architectural choices make large models more tractable to study and that interpretability difficulty is not a fixed consequence of model size.

Submission Number: 280