Causal Influence Detection for Improving Efficiency in Reinforcement Learning

Maximilian Seitzer, Bernhard Schölkopf, Georg Martius

Keywords: reinforcement learning, causal inference, exploration, intrinsic motivation, prioritized replay

Abstract: Many reinforcement learning (RL) environments consist of independent entities that interact sparsely. In such environments, RL agents have only limited influence over other entities in any particular situation. Our idea in this work is that learning can be efficiently guided by knowing when and what the agent can influence with its actions. To achieve this, we introduce a measure of situation-dependent causal influence based on conditional mutual information and show that it can reliably detect states of influence. We then propose several ways to integrate this measure into RL algorithms to improve exploration and off-policy learning. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We propose a method of detecting the causal influence of RL agents on the environment and use it to improve the sample efficiency of RL algorithms.

Code: https://github.com/martius-lab/cid-in-rl