Offline Model-Based Reinforcement Learning with Causal Structure

Zhengmao Zhu

Abstract:

Model-based methods have recently been shown promising for offline reinforcement learning (RL), which aims at learning good policies from historical data without interacting with the environment.
Previous model-based offline RL methods employ a straightforward prediction method that maps the states and actions directly to the next-step states.
However, such a prediction method tends to capture spurious relations caused by the sampling policy preference behind the offline data.
It is sensible that the environment model should focus on causal influences, which can facilitate learning an effective policy that can generalize well to unseen states.
In this paper, we first provide theoretical results that causal environment models can outperform plain environment models in offline RL by incorporating the causal structure into the generalization error bound.
We also propose a practical algorithm, o\textbf{F}fline m\textbf{O}del-based reinforcement learning with \textbf{C}a\textbf{U}sal \textbf{S}tructure (FOCUS), to illustrate the feasibility of learning and leveraging causal structure in offline RL.
Experimental results on two benchmarks show that FOCUS reconstructs the underlying causal structure accurately and robustly, and, as a result, outperforms both model-based offline RL algorithms and causal model-based offline RL algorithms.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)