Platonic Transformers: A Solid Choice for Equivariance

Mohammad Mohaiminul Islam, Rishabh Anand, David R Wessels, Friso de Kruiff, Thijs P. Kuipers, Rex Ying, Clara I. Sánchez, Sharvaree Vadgama, Georg Bökman, Erik J Bekkers

Track: tiny paper (up to 4 pages)

Keywords: equivariant transformers, scalable equivariance, platonic solids

Abstract:

Transformers lack inductive biases for geometric symmetries common across
scientific domains. Existing equivariant methods often sacrifice the efficiency and
flexibility that make Transformers so scalable through complex, computationally
intensive designs. We introduce the *Platonic Transformer* to resolve this long-
standing trade-off. By defining attention relative to reference frames from Platonic
solid symmetry groups, our method induces a principled weight-sharing scheme.
This enables combined E(3) equivariance to continuous translations and Platonic
symmetries, while preserving the exact architecture and computational cost of a
standard Transformer. Platonic Transformers achieve highly competitive results on
molecular property prediction and unconditional generation tasks (QM9, OMol25),
leveraging geometric constraints at no additional cost.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 100