Why Equivariant Networks Lose Information: Invariant Rings and the Role of Aggregation

TMLR Paper6979 Authors

Abstract: Equivariant neural networks exhibit fundamental expressivity limitations: rotation-equivariant networks collapse directional information to radial features, and matrix-equivariant networks show rank degeneracy. We explain these phenomena using classical invariant theory and prehomogeneous vector space (PVS) theory. For $\mathrm{SO}(3)$ on $\mathbb{R}^3$, the First Fundamental Theorem forces equivariant maps to be radial scalings; for $\mathrm{GL}(n) \times \mathrm{GL}(n)$ on matrices, PVS theory shows the invariant ring contains only constants. Our central finding is that aggregation, not depth, escapes these constraints: product representations $V^n$ have richer invariant rings with cross-invariants (e.g., dot products encoding angles) inaccessible to single-fiber processing. We connect this theory to modern architectures---SchNet, PaiNN, DimeNet, MACE---showing their body-order corresponds to which $V^n$ they access. Experiments confirm that $\mathrm{SO}(3)$- versus $\mathrm{O}(3)$-invariant networks exhibit categorically different expressivity on pseudoscalar targets ($R^2 = 1.00$ vs. $R^2 < 0$), and that cross-invariants enable learning angles while norm-only features cannot. These results provide design guidance: prioritize multi-body interactions over depth when expressivity is limited.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Jean_Barbier2

Submission Number: 6979