Multiplication Beyond Groups: Stratified Fourier Mechanisms in Transformer Circuits

Zitong Chen, Junaid Hasan, Akhil Srinivasan, Hemkesh Bandi, Jarod Alper

Keywords: Methods (probing, steering, causal interventions), Circuit Analysis, Attribution Graphs, Feature Geometry

TL;DR: Transformers trained on composite modular multiplication extend group-based Fourier mechanisms by stratifying computation into local regions where local inverses and character features explain logits.

Abstract: Transformers have demonstrated a remarkable ability to learn algorithmic reasoning, yet mechanistic analyses have mostly focused on globally invertible operations such as cyclic addition and group composition. In this work, we investigate how small transformers learn modular integer multiplication over composite moduli, a fundamentally non-invertible operation due to the presence of zero-divisors. We propose the monoid extension: a localized generalization of Group Composition via Representation (GCR) that suggests the learned computation does not rely on a single global representation space. Instead, the model partitions the input space into local hierarchical algebraic regions, where group-like structure survives and Fourier mechanisms can be applied. In transformers trained on square-free modular multiplication, we find that embeddings organize around these regions, attention exhibits class-sensitive routing and low-rank write directions, and local character features explain a large fraction of the model's output logits. Our results suggest that representation-theoretic mechanisms previously identified for group operations can extend beyond groups to more general structures.

Submission Number: 661