R3: Robust Rubric-Agnostic Reward Models

David Anugraha, Zilu Tang, Lester James Validad Miranda, Hanyang Zhao, Shou-Yi Hung, Mohammad Rifqi Farhansyah, Garry Kuwanto, Derry Tanti Wijaya, Genta Indra Winata

Keywords: reward model, LLM, rubric, evaluation, generation models

TL;DR: We introduce R3, a reward modeling framework that is rubric-agnostic, generalizable, and interpretable, enabling more transparent and flexible alignment of language models with human preferences.

Abstract: Reward models are essential for aligning language model outputs with human preferences, yet existing approaches often lack both controllability and interpretability. These models are typically optimized for narrow objectives, limiting their generalizability to broader downstream tasks. Moreover, their scalar outputs are difficult to interpret without contextual reasoning. To address these limitations, we introduce R3, a novel reward modeling framework that is rubric-agnostic, generalizable across evaluation dimensions, and provides interpretable, reasoned score assignments. R3 enables more transparent and flexible evaluation of language models, supporting robust alignment with diverse human values and use cases. Our models, data, and code will be available as open source.

Submission Number: 230