The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning

Zhanke Zhou, Xiangyu Lu, Chentao Cao, Brando Miranda, Tongliang Liu, Bo Han, Sanmi Koyejo

TL;DR: We propose CoDaPO, a confidence and difficulty–adaptive RL framework for LLM reasoning

Abstract: RL with verifiable rewards can substantially improve LLM reasoning, yet standard GRPO-style training often treats easy, hard, and learnable questions alike through uniform sampling and weighting, leading to inefficient compute allocation. We study GRPO by tracking token log-probabilities, group-normalized advantages, and the induced token-level update weights. This reveals three recurring dynamics as training proceeds: (1) confidence inflation, (2) advantage contraction, and (3) hierarchical convergence. These findings suggest that the utility of each update depends strongly on both question difficulty and the model’s current competence. Motivated by this, we propose Confidence and Difficulty-adaptive Policy Optimization (CoDaPO), which assigns each question a bounded value from rollout confidence and empirical difficulty. CoDaPO then uses this value to reweight policy updates and resample high-value learnable questions within mini-batches, thereby increasing discovery within the learnable band under a fixed compute budget. Across twelve benchmarks, CoDaPO consistently improves accuracy over existing RL methods. Our code is publicly available at https://github.com/tmlr-group/CoDaPO.

Lay Summary:

Large language models can be taught to reason, such as solving math or writing code, by letting them practice and rewarding correct answers. But standard training treats every practice question alike, whether the model has already mastered it, finds it nearly impossible, or sits in between. This wastes effort. The model keeps drilling questions it has already solved, growing overconfident even in its mistakes, while truly hard questions barely improve because it so rarely stumbles onto a correct answer to learn from.

We developed CoDaPO, which judges how much can still be learned from each question, using how confident the model is and how often it currently answers correctly. It then focuses the limited training effort on the "learnable" questions, hard enough to be worth the work yet tractable enough that success happens, and gives them extra attempts to raise the chance of discovering a correct solution.

Using no extra computing power, CoDaPO improves accuracy across twelve math, science, and coding benchmarks.

Link To Code: https://github.com/tmlr-group/CoDaPO

Primary Area: Deep Learning->Large Language Models

Keywords: Reinforcement learning, Large languge models, Reasoning

Submission Number: 24524