Explainable Rewards in RLHF Using LLM-as-a-Judge

Yiran Shen, Aditya Emmanuel Arokiaraj John, Brandon Fain

Keywords: Large Language Models; Reinforcement Learning from Human Feedback; Explainability

Abstract: Reinforcement Learning from Human Feedback (RLHF) has been gaining popularity as a method for aligning Large Language Models (LLMs) with human preferences. It involves performing Supervised Fine-Tuning (SFT) followed by fine-tuning using a reward model trained on human preference data. However, two primary issues with this approach are the difficult and expensive curation of human preference data and the opaque, black-box nature of the rewards. To address these issues, this paper introduces a novel framework for aligning LLMs with human preferences. Our framework involves using representative sub-dimensions for specific tasks to generate rewards by leveraging a performant out-of-the-box LLM. We evaluate our approach by fine-tuning two models, one using our approach and one using traditional black-box rewards. Evaluation using an advanced LLM-based method demonstrates that our approach maintains the performance of the black-box baseline while offering superior explainability and flexibility. This framework not only enhances transparency in RLHF but also eliminates reliance on expensive human-curated preference data.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 13365