Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages

Andy Yang, David Chiang, Dana Angluin

Keywords: Transformers, Formal Language Theory, Logic, Automata, Expressivity

TL;DR: We prove masked hard-attention transformers recognize exactly the star-free regular languages.

Abstract: The expressive power of transformers over inputs of unbounded size can be studied through their ability to recognize classes of formal languages. In this paper, we establish exact characterizations of transformers with hard attention (in which all attention is focused on exactly one position) and attention masking (in which each position only attends to positions on one side). With strict masking (each position cannot attend to itself) and without position embeddings, these transformers are expressively equivalent to linear temporal logic (LTL), which defines exactly the star-free languages. A key technique is the use of Boolean RASP as a convenient intermediate language between transformers and LTL. We then take numerous results known for LTL and apply them to transformers, showing how position embeddings, strict masking, and depth all increase expressive power.

Primary Area: Deep learning architectures

Submission Number: 18519