Towards Foundation Models with Mathematical Understanding

Peter Belcak, Roger Wattenhofer

Keywords: transformer model, mathematical understanding, mathematical concepts, integer sequences, FACT

TL;DR: We investigate the ability of transformer models to build representations of integer sequences that are of utility to tasks where deeper mathematical understanding is needed, and find that they are capable of doing so under multi-task pre-training.

Abstract:

We investigate the ability of transformer models to build representations of integer sequences that are of utility to tasks where deeper mathematical understanding is needed.
To that end, we train BERT-like transformer encoders to assess the impact of individual pre-training tasks on the quality of the resulting model, and evaluate them for sequence classification, continuation, unmasking, complexity prediction, and next sequence-part prediction.
We find that the models both outperform benchmark baselines and provide reasonable estimates of the complexity of the mathematical rules behind the sequences.