Subspace Inference Enables Active Preference based Learning of Neural Network Reward Models

ICLR 2026 Conference Submission20714 Authors

Keywords: Bayesian neural network, reinforcement learning from human feedback, preference-based reward modeling, active learning, human-in-the-loop learning

TL;DR: Sample- and compute-efficient method for training Bayesian neural networks, to enable Bayesian active learning of reward models from human preferences.

Abstract:

Reinforcement learning from human feedback (RLHF) has emerged as a powerful approach for aligning decision-making agents with human intentions, primarily through the use of reward models trained on human preferences. However, RLHF suffers from poor sample efficiency, as each feedback provides minimal information, making it necessary to collect large amounts of human feedback. Active learning addresses this by enabling agents to select informative queries, but effective uncertainty quantification required for active learning remains a challenge. While ensemble methods and dropout are popular for their simplicity, they are computationally expensive at scale and do not always provide good posterior approximation.
Inspired by the recent advances in approximate Bayesian inference, we develop a method that leverages Bayesian filtering in neural network subspaces to efficiently maintain model posterior for active reward modeling. Our approach enables scalable sampling of neural network reward models to efficiently compute active learning acquisition functions. Experiments on the D4RL benchmark demonstrate that our approach achieves superior sample efficiency, scalability, and calibration compared to ensemble methods and dropout, and leads to competitive offline reinforcement learning policy performance. This highlights the potential of scalable Bayesian methods for preference-based reward modeling in RLHF.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 20714