LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning

Yuhuai Wu, Markus Norman Rabe, Wenda Li, Jimmy Ba, Roger Baker Grosse, Christian Szegedy

Keywords: Theorem proving, Pre-training, Inductive bias, Reasoning.

Abstract: While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks. Here, we replace architecture engineering by encoding inductive bias in the form of datasets. Inspired by Peirce's view that deduction, induction, and abduction form an irreducible set of reasoning primitives, we design three synthetic tasks that are intended to require the model to have these three abilities. We specifically design these synthetic tasks in a way that they are devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks. This defines a new pre-training methodology called ``"LIME" (Learning Inductive bias for Mathematical rEasoning). Models trained with LIME significantly outperform vanilla transformers on three very different large mathematical reasoning benchmarks. Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We designed inductive bias of mathematical reasoning in the form of dataset and use pretraining to learn inductive biases, demonstrating significant gains over three mathematical reasoning datasets.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2101.06223/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=8xVsXPCdm3