Can Differentiable Decision Trees Learn Interpretable Reward Functions?
Primary Area: reinforcement learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Reinforcement Learning, Human-AI Alignment, Explainability, Reward Functions, Interpretability, RLHF, Visualization or interpretation of learned representations
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: Modeling reward functions as differentiable decision trees enables learning interpretable and expressive reward functions.
Abstract: There is an increasing interest in learning reward functions that model human
intent and human preferences. However, many frameworks use blackbox learning
methods that, while expressive, are difficult to interpret. We propose and evaluate
a novel approach for learning expressive and interpretable reward functions from
preferences using Differentiable Decision Trees (DDTs). Our experiments across
several domains, including Cartpole, Visual Gridworld environments and Atari
games, provide evidence that that the tree structure of our learned reward function is
useful in determining the extent to which the reward function is aligned with human
preferences. We experimentally demonstrate that using reward DDTs results in
competitive performance when compared with larger capacity deep neural network
reward functions. We also observe that the choice between soft and hard (argmax)
output of reward DDT reveals a tension between wanting highly shaped rewards
to ensure good RL performance, while also wanting simpler, more interpretable
rewards.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
Supplementary Material: pdf
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 6859
Loading