Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Proving

Yong Lin, Shange Tang, Bohan Lyu, Jiayun Wu, Hongzhou Lin, Kaiyu Yang, Jia LI, Mengzhou Xia, Danqi Chen, Sanjeev Arora, Chi Jin

Keywords: Formal reasoning, verification, Lean, self-improvement

TL;DR: Introduce Goedel-Prover, an open-source language model that achieves SOTA in automated theorem proving in Lean

Abstract:

We introduce Goedel-Prover, an open-source language model that achieves state-of-the-art performance in automated formal proof generation for mathematical problems. A key challenge in this field is the scarcity of formalized mathematical statements and proofs, which we address through the following approaches. First, we train statement formalizers to translate natural language math problems from Numina into the formal language Lean 4, and use an LLM to verify that the formal statements accurately preserve the content of the original problems. This results in a dataset of 1.64 million formal statements. We then iteratively build a large dataset of formal proofs by training a series of provers: each prover is able to prove many statements that the previous ones could not, and these new proofs are added to the training set for the next prover. Despite using only supervised fine-tuning, our final prover (fine-tuned on DeepSeek-Prover-V1.5-base) significantly outperforms the previous best open-source model, DeepSeek-Prover-V1.5, which uses reinforcement learning.
On the MiniF2F benchmark, our model achieves a success rate of 57.6\% (Pass@32), surpassing DeepSeek-Prover-V1.5 by 7.6\%.
On PutnamBench, Goedel-Prover successfully solves 7 problems (Pass@512), ranking first on the leaderboard.
Furthermore, it generates 29.7K formal proofs for Lean Workbook problems, nearly doubling the 15.7K produced by prior work.
We provide extensive discussion of our training methodology, highlighting the key design choices that contribute to Goedel-Prover’s strong performance. Finally, we explore reinforcement learning on top of Goedel-Prover-SFT, offering insights into its potential benefits and limitations.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Submission Number: 919