Mathematical Reasoning via Self-supervised Skip-tree Training

Markus Norman Rabe, Dennis Lee, Kshitij Bansal, Christian Szegedy

Keywords: self-supervised learning, mathematics, reasoning, theorem proving, language modeling

Abstract: We demonstrate that self-supervised language modeling applied to mathematical formulas enables logical reasoning. To measure the logical reasoning abilities of language models, we formulate several evaluation (downstream) tasks, such as inferring types, suggesting missing assumptions and completing equalities. For training language models for formal mathematics, we propose a novel skip-tree task. We find that models trained on the skip-tree task show surprisingly strong mathematical reasoning abilities, and outperform models trained on standard skip-sequence tasks. We also analyze the models' ability to formulate new conjectures by measuring how often the predictions are provable and useful in other proofs.

One-sentence Summary: We demonstrate that self-supervised language modeling applied to mathematical formulas enables logical reasoning.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [HOList](https://paperswithcode.com/dataset/holist)