Lemmanaid: Neuro-Symbolic Lemma Conjecturing

Yousef Alhessi, Sólrún Halla Einarsdóttir, George Granberry, Emily First, Moa Johansson, Sorin Lerner, Nicholas Smallbone

Keywords: conjecturing, theorem proving, formalization, neuro-symbolic, proof assistants, interactive theorem proving, AI for math

TL;DR: We combine language models and symbolic methods to conjecture lemmas for mathematical formalization.

Abstract: Automatically conjecturing useful, interesting and novel lemmas would greatly improve automated reasoning tools and lower the bar for formalizing mathematics in proof assistants. It is however a very challenging task for both neural and symbolic approaches. We present the first steps towards a practical neuro-symbolic lemma conjecturing tool, LEMMANAID, that combines Large Language Models (LLMs) and symbolic methods, and evaluate it on proof libraries for the Isabelle proof assistant. We train an LLM to generate lemma templates that describe the shape of a lemma, and use symbolic methods to fill in the details. We compare LEMMANAID against an LLM trained to generate complete lemma statements as well as previous fully symbolic conjecturing methods. LEMMANAID outperforms both neural and symbolic methods on test sets from Isabelle's HOL library and from its Archive of Formal Proofs, discovering between 29-39.5\% of the gold standard human written lemmas. This is 8-15\% more lemmas than the neural-only method. By leveraging the best of both symbolic and neural methods we can generate useful lemmas for a wide range of input domains, facilitating computer-assisted theory development and formalization.

Submission Number: 110