Stable Reinforcement Learning for Efficient Reasoning

Mz Dai, Shixuan Liu, Qingyi Si

Keywords: Large language model, Reasoning model, Reinforcement learning, Post-training

Abstract: Recently, reinforcement learning (RL) methods like GRPO have drawn the LLM community's attention. However, such rule-based 0/1 outcome reward methods lack the capability to regulate the intermediate reasoning processes during chain-of-thought (CoT) generation, leading to severe overthinking phenomena. In response, recent studies have designed reward functions to reinforce models' behaviors in producing shorter yet correct completions. Nevertheless, we observe that these length-penalty reward functions exacerbate RL training instability: as the completion length decreases, model accuracy abruptly collapses, often occurring early in training. To address this issue, we propose a simple yet effective solution GRPO-λ, an efficient and stabilized variant of GRPO, which dynamically adjusts the reward strategy by monitoring the correctness ratio among completions within each query-sampled group. A low correctness ratio indicates the need to avoid length penalty that compromises CoT quality, triggering a switch to length-agnostic 0/1 rewards that prioritize reasoning capability. A high ratio maintains length penalties to boost efficiency. Experimental results show that our approach avoids training instability caused by length penalty while maintaining the optimal accuracy-efficiency trade-off. On five popular reasoning benchmarks, it improves average accuracy by 0.36% ~ 3.76% while reducing CoT sequence length by 44.2% ~ 62.3%.

Submission Number: 142