The Geometry and Topology of Modular Addition Representations

Gabriela Moisescu-Pareja, Gavin McCracken, Colin Daniels, Harley Wiltzer, Vincent Létourneau, Jonathan Love

Track: Extended Abstract Track

Keywords: geometry, topology, representation learning, mechanistic interpretability, toy models, group theory, modular addition, universality

Abstract: The Clock and Pizza interpretations, associated with neural architectures differing in either uniform or learnable attention, were introduced to argue that different architectural designs can yield distinct circuits for modular addition. Applying geometric and topological analyses to learned representations, we show that this is not the case: Clock and Pizza circuits are topologically and geometrically equivalent and are thus equivalent representations.

Submission Number: 131