Evaluating Agents Without Rewards

Brendon Matusch, Jimmy Ba, Danijar Hafner

Keywords: reinforcement learning, task-agnostic, agent evaluation, exploration, information gain, empowerment, curiosity

Abstract: Reinforcement learning has enabled agents to solve challenging control tasks from raw image inputs. However, manually crafting reward functions can be time consuming, expensive, and prone to human error. Competing objectives have been proposed for agents to learn without external supervision, such as artificial input entropy, information gain, and empowerment. Estimating these objectives can be challenging and it remains unclear how well they reflect task rewards or human behavior. We study these objectives across seven agents and three Atari games. Retrospectively computing the objectives from the agent's lifetime of experience simplifies accurate estimation. We find that all three objectives correlate more strongly with a human behavior similarity metric than with task reward. Moreover, input entropy and information gain both correlate more strongly with human similarity than task reward does.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2012.11538/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=YCIaz4-Z-G