OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment

ACL ARR 2026 January Submission7802 Authors

Keywords: reward modeling, large language model, rubric

Abstract:

Reward modeling lies at the core of reinforcement learning from human feedback (RLHF), yet most existing reward models rely on scalar or pairwise judgments that fail to capture the multifaceted nature of human preferences.
Recent studies have explored \emph{rubrics-as-rewards} (RaR) that uses structured criteria to capture multiple dimensions of response quality. However, producing rubrics that are both reliable and scalable remains a key challenge.
In this work, we introduce OpenRubrics, a diverse, large-scale collection of (prompt, rubric) pairs for training rubric-generation and rubric-based reward models.
To elicit discriminative and comprehensive evaluation signals, we introduce \emph{Contrastive Rubric Generation} (CRG), which derives both hard rules (explicit constraints) and principles (implicit qualities) by contrasting preferred and rejected responses. We further remove noisy rubrics via preserving preference–label consistency. Across multiple reward-modeling benchmarks, our rubric-based reward model, Rubric-RM, surpasses strong size-matched baselines by 8.4\%. These gains transfer to policy models on instruction-following and biomedical benchmarks.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: data augmentation, reinforcement learning

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 7802