Linear algebra with transformers

Francois Charton

Keywords: Mathematics, Transformers, Computation, Numerical, Linear algebra

Abstract: Most applications of transformers to mathematics, from integration to theorem proving, focus on symbolic computation. In this paper, we show that transformers can be trained to perform numerical calculations with high accuracy. We consider problems of linear algebra: matrix transposition, addition, multiplication, eigenvalues and vectors, singular value decomposition, and inversion. Training small transformers (up to six layers) over datasets of random matrices, we achieve high accuracies (over 90%) on all problems. We also show that trained models can generalize out of their training distribution, and that out-of-domain accuracy can be greatly improved by working from more diverse datasets (in particular, by training from matrices with non-independent and identically distributed coefficients). Finally, we show that few-shot learning can be leveraged to retrain models to solve larger problems.

One-sentence Summary: We train transformers to solve problems of linear algebra (eigenvalues, matrix inversion), with high accuracy

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2112.01898/code)