Exploiting Action Distances for Reward Learning from Human Preferences

Mudit Verma, Siddhant Bhambri, Subbarao Kambhampati

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Preference based Reinforcement Learning, Human Aware AI, Reward Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We use the joint policy being learned in PbRL to obtain the proposed action distance measure which when used as an additional auxiliary task to the reward model significantly improves performance and accelerates policy learning.

Abstract: Preference-based Reinforcement Learning (PbRL) with binary preference feedbacks over trajectory pairs has proved to be quite effective in learning complex preferences of a human in the loop in domains with high dimensional state spaces and action spaces. While the human preference is primarily inferred from the feedback provided, we propose that the policy being learned (jointly with the reward model) during training can provide valuable learning signal about the structure of the state space that can be leveraged by the reward learning process. We introduce an action distance measure based on the policy and use it as an auxiliary prediction task for reward learning to influence its embedding space. This measure not only provides insight into the transition dynamics of the environment but also informs about the reachability of states and the overall state space structure. We evaluate the performance and sample efficiency of our approach using a combination of six tasks in Meta-World domains with simulated oracles. We also conduct human in the loop evaluation on three tasks to confirm our findings from oracular experiments. We demonstrate that the proposed simple auxiliary task for constraining reward model's embedding space can provide strong empirical improvements to sample efficiency and accelerate policy learning.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8628