Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators

ICLR 2026 Conference Submission20603 Authors

Keywords: Reinforcement learning, Reinforcement learning benchmarks, offline reinforcement learning, sim2real gap

Abstract: In many reinforcement learning (RL) applications one cannot easily let the agent act in the world; this is true for autonomous vehicles, healthcare applications, and even some recommender systems, to name a few examples. Offline RL provides a way to train agents without real-world exploration, but is often faced with biases due to data distribution shifts, limited coverage, and incomplete representation of the environment. To address these issues, practical applications have tried to combine simulators with grounded offline data, using so-called hybrid methods. However, constructing a reliable simulator is in itself often challenging due to intricate system complexities as well as missing or incomplete information. In this work, we outline four principal challenges for combining offline data with imperfect simulators in RL: simulator modeling error, partial observability, state and action discrepancies, and hidden confounding. To help drive the RL community to pursue these problems, we construct ``Benchmarks for Mechanistic Offline Reinforcement Learning'' (B4MRL), which provide dataset-simulator benchmarks for the aforementioned challenges. Our results show that current algorithms fail to synergize these sources, often performing worse than using one source alone, especially when faced with hidden confounding.

Primary Area: reinforcement learning

Submission Number: 20603