Symbolic World Models in Lean 4 for Reinforcement Learning

Matěj Kripner

Keywords: model-based RL, symbolic regression, Lean, evolutionary algorithm

TL;DR: Combining symbolic model-based RL and formal mathematics languages.

Abstract: We propose a novel approach to model-based reinforcement learning by synthesizing symbolic world models in the Lean 4 proof assistant. Leveraging Lean's formal language for mathematics, we encode environment dynamics as interpretable, verifiable rules. Our system integrates a planning agent, an evolutionary algorithm inspired by AlphaEvolve, and a Lean server that predicts the dynamics of an environment using a set of already synthesized rules. We evaluate our approach on a custom cellular automaton environment called FireHelicopter. This environment simulates the dynamics of a forest fire and requires the agent to maximize forest preservation. We explore two training objectives: a pragmatic one focused on maximizing agent's return, and a descriptive one prioritizing accurate world prediction. To our knowledge, this is the first use of a general formal mathematics language for model-based RL. We hypothesize that this is a promising avenue for sample-efficient, safe, and interpretable reinforcement learning in real-world scenarios.

Format: We have read the camera-ready instructions, and our paper is formatted with the provided template.

De-Anonymization: This submission has been de-anonymized.

Presenter: ~Matěj_Kripner1

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 8