Composing Linear Layers from Irreducibles

Travis Pence, Daisuke Yamada, Vikas Singh

Keywords: Clifford Algebra, Compositions, Modular

TL;DR: We show that linear layers, the primary player in large language models, can be composed from fundamental algebraic primitives with exponentially less parameters.

Abstract:

Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood.
We investigate this compositional structure in linear layers by asking: \textit{can we identify/synthesize linear transformations from a minimal set of geometric primitives?}
Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors---geometric objects encoding oriented planes---and introduce a differentiable algorithm that decomposes them into products of rotors.
This construction uses only $\mathcal{O}(\log^2 d)$ parameters, versus $\mathcal{O}(d^2)$ required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, our rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 17467