Advancing LLM Reasoning with Natural Language and Numerical Feedback

Xiaoying Zhang, Hao Sun, Yipeng Zhang, Kaituo Feng, Chaochao Lu, Chao Yang, Helen M. Meng

Keywords: LLM reasoning, reinforcement learning, natural language feedback

TL;DR: We propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for effective policy optimization in LLM reasoning.

Abstract: Recent advances in reinforcement learning (RL) with numerical feedback, such as scalar rewards, have significantly enhanced the complex reasoning capabilities of large language models (LLMs). Despite this success, we identify three key challenges encountered by RL with solely numerical feedback: performance plateaus, limited effectiveness of spontaneous self-reflection, and persistent failures. We then demonstrate that RL-finetuned models, even after exhibiting performance plateaus, can generate correct refinements on persistently failed problems by leveraging natural language feedback in the form of critiques. Building on this insight, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for effective policy optimization. Critique-GRPO enables LLMs to learn from initial responses and critique-guided self-refinements simultaneously while maintaining exploration. Additionally, we employ a shaping function to amplify learning from correct, especially unfamiliar, refinements and penalize incorrect ones. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0\%, +21.6\%, and +15.0\% on Qwen2.5-7B-Base, Qwen2.5-Math-7B-Base, and Qwen3-8B across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7\% Pass@1 improvement on AIME 2024.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 7790