Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning

ACL ARR 2025 May Submission6834 Authors

Abstract: Since DeepSeek-R1 popularized, Group Relative Policy Optimization (GRPO) has become the core part of training Reasoning LLMs. However, we find some deficiency that influences RL stability and inference efficiency, like zero-variance in advantage estimation. Thus, we propose Adaptive Group Policy Optimization (AGPO) which contains a simple but effective modification: a revised objective function to mitigate training fluctuation and zero advantage. The experiments demonstrate our method achieves more stable training and superior performance with significantly fewer tokens in reasoning steps.

Paper Type: Short

Research Area: Language Modeling

Research Area Keywords: chain-of-thought, continual learning

Languages Studied: English

Submission Number: 6834