GAR: Generative Adversarial Reinforcement Learning for Formal Theorem Proving

Ruida WANG, Jiarui Yao, Rui Pan, Shizhe Diao, Tong Zhang

Keywords: Lean4, Reinforcement Learning, LLM

TL;DR: This paper proposes GAR, a comprehensive RL training famework for Lean4 prover training that enables more efficient and effective RL training by adversarial training of both statement composer and theorem prover.

Abstract: Solving math problems through verifiable languages such as Lean has significantly impacted both the mathematics and computer science communities. Current state-of-the-art models are often trained with expensive online Reinforcement Learning (RL) or expert iteration. However, these approaches rely on fixed problem sets, which causes inefficient training and limits the model to tackle complex problems. To overcome these limitations, we propose **GAR**: *Generative Adversarial Reinforcement learning*, a comprehensive RL training framework that jointly trains the problem composer and solver in an adversarial loop. **GAR** introduces an implicit curriculum learning mechanism, which aligns task difficulty with the prover's evolving capability. It thereby improves the training efficiency and enables stronger performance of proving advanced theorems. Experiments show that with **GAR** training, Goedel-Prover-V2-8B and DeepSeek-Prover-V2-7B achieve an average relative improvement in pass@32 of **4.20%** on MiniF2F-Test benchmark, while DeepSeek-Prover-V2's pass@32 on ProofNet-Test increases from 22.58% to **25.81%**. Beyond formal proving, **GAR** establishes a general RL paradigm for co-evolution of problem generation and solving under verifiable environments. The training code for this paper is open-sourced in [https://github.com/RickySkywalker/GAR-Official](https://github.com/RickySkywalker/GAR-Official)

Primary Area: foundation or frontier models, including LLMs

Submission Number: 7861