ACAMDA: Improving Data Efficiency in Reinforcement Learning Through Guided Counterfactual Data Augmentation

Yuewen Sun, Erli Wang, Biwei Huang, Chaochao Lu, Lu Feng, Changyin Sun, Kun Zhang

Abstract:

Data augmentation plays a crucial role in improving the data efficiency of reinforcement learning (RL). However, the generation
of high-quality augmented data remains a significant challenge. To overcome this, we introduce ACAMDA (Adversarial
Causal Modeling for Data Augmentation), a novel framework that integrates two causality-based tasks: causal
structure recovery and counterfactual estimation. The unique aspect of ACAMDA lies in its ability to recover temporal
causal relationships from limited non-expert datasets. The identification of the sequential cause-and-effect allows the
creation of realistic yet unobserved scenarios. We utilize this characteristic to generate guided counterfactual datasets,
which, in turn, substantially reduces the need for extensive data collection. By simulating various state-action pairs under
hypothetical actions, ACAMDA enriches the training dataset for diverse and heterogeneous conditions. Our experimental
evaluation shows that ACAMDA outperforms existing methods, particularly when applied to novel and unseen domains.