Entropy Scheduling in Reinforcement Learning for Large Language Models

Xingjin Wang, Howe Tissue, Lu Wang, Linjing Li, Daniel Dajun Zeng

Keywords: Reinforcement Learning with Verifiable Rewards; Entropy Scheduling

Abstract: We observe that __entropy__ in reinforcement learning functions analogously to the __learning rate__ in LLMs. Maintaining stable entropy, as demonstrated in DAPO, helps stabilize RL training, while rapid entropy annealing (i.e., so-called entropy collapse) accelerates local performance improvement and enables faster convergence. We argue that these two processes are not antithetical, but can be effectively controlled and scheduled within a single training run, similar to learning rate scheduling. We propose **E**ntropy **S**chduling **(ES)**, which optimizes different pre-set goals (e.g. $k$ in optimizing Pass@$k$) by controlling and scheduling entropy at each step of the RL process. We find that maintaining stable entropy early in training followed by entropy annealing achieves superior performance. Moreover, since stable-state entropy and annealed entropy exhibit distinctly different learning dynamics, curriculum learning can be seamlessly integrated to maximize model performance based on different entropy phases. We show that entropy scheduling is straightforward to implement and intuitive in design. Extensive experiments suggest that it delivers consistent and stable performance improvements across diverse models and algorithms.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 5177