Out-of-distributon Tests Reveal Compositionality in Chess Transformers

ICLR 2026 Conference Submission21232 Authors

Keywords: Language model, transformer, chess model, out-of-distribution generalization, rule extrapolation, chess960

Abstract:

Chess is a canonical example of a task that requires rigorous reasoning and long-term planning. Modern decision Transformers - trained similarly to LLMs - are able to learn competent gameplay, but it is unclear to what extent they truly capture the rules of chess.
To investigate this, we train a 270M parameter chess Transformer and test it on out-of-distribution scenarios, designed to reveal failures of systematic generalisation. Our analysis shows that Transformers exhibit compositional generalisation, as evidenced by strong rule extrapolation: they adhere to fundamental ‘syntactic’ rules of the game by consistently choosing valid moves even in situations very different from the training data. Moreover, they also generate high-quality moves for OOD puzzles. In a more challenging test, we evaluate the models on variants including Chess960 (Fischer Random Chess) - a variant of chess where starting positions of pieces are randomised. We found that while the model exhibits basic strategy adaptation, they are inferior to symbolic AI algorithms that perform explicit search, but gap is smaller when playing against users on Lichess. Moreover, the training dynamics revealed that the model initially learns to move only its own pieces, suggesting an emergent compositional understanding of the game.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 21232