Curriculum-RLAIF: Curriculum Alignment with Reinforcement Learning from AI Feedback

ICLR 2026 Conference Submission13414 Authors

Keywords: RLAIF, Reinforcement Learning, LLM Alignment, Curriculum Learning, RLHF

Abstract: Reward models trained with conventional Reinforcement Learning from AI Feedback (RLAIF) methods suffer from limited generalizability, which hinders the alignment performance of the policy model during reinforcement learning (RL). This challenge stems from various issues, including distribution shift, preference label noise, and mismatches between overly challenging samples and model capacity. In this paper, we attempt to enhance the generalizability of reward models through a data-centric approach, driven by the insight that these issues are inherently intertwined from the perspective of data difficulty. To address this, we propose a novel framework, $\textit{Curriculum-RLAIF}$, which constructs preference pairs with varying difficulty levels and produces a curriculum that progressively incorporates preference pairs of increasing difficulty for reward model training. Our experimental results suggest that reward models trained with Curriculum-RLAIF achieve improved generalizability, significantly increasing the alignment performance of the policy model by a large margin without incurring additional inference costs compared to various non-curriculum baselines. Detailed analysis and comparisons with alternative approaches, including data selection via pretrained reward models ond self-selection mechanisms demonstrate the superiority of our approach in terms of simplicity, efficiency, and effectiveness.

Primary Area: reinforcement learning

Submission Number: 13414