Transformers are Inherently Succinct

ICLR 2026 Conference Submission18954 Authors

Keywords: Transformers, complexity, expressivity, automata, LTL

Abstract:

We propose succinctness as a measure of expressive power
of a transformer in describing a concept. To this end, we prove that
transformers are highly expressive in that they can represent formal
languages substantially more succinctly than standard representations of
formal languages like finite automata and Linear Temporal Logic (LTL)
formulas. As a by-product of this expressivity, verifying
even simple properties of transformers is shown to be provably intractable (i.e. EXPSPACE-complete).

Primary Area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Submission Number: 18954