Learning the greatest divisor - Explainable predictions in transformers

Published: 28 Oct 2023, Last Modified: 16 Nov 2023MATH-AI 23 PosterEveryoneRevisionsBibTeX
Keywords: Transformers, arithmetic, explainability
TL;DR: Transformers can learn to predict greatest common divisors. Their predictions are fully explainable. A log-uniform distribution of operands and outcomes achieves best results.
Abstract: We train small transformers to calculate the greatest common divisor (GCD) of two positive integers, and show that their predictions are fully explainable. During training, models learn a list $\mathcal D$ of divisors, and predict the largest element of $\mathcal D$ that divides both inputs. We also show that training distributions have a large impact on performance. Models trained from uniform operands only learn a handful of GCD (up to $38$ out of $100$). Training from log-uniform operands boosts performance to $73$ correct GCD, and training from a log-uniform distribution of GCD to $91$.
Submission Number: 8