Causal Inference Q-Network: Toward Resilient Reinforcement Learning

Chao-Han Huck Yang, Danny I-Te Hung, Yi Ouyang, Pin-Yu Chen

Keywords: Deep Reinforcement Learning, Causal Inference, Robust Reinforcement Learning, Adversarial Robustness

Abstract: Deep reinforcement learning (DRL) has demonstrated impressive performance in various gaming simulators and real-world applications. In practice, however, a DRL agent may receive faulty observation by abrupt interferences such as black-out, frozen-screen, and adversarial perturbation. How to design a resilient DRL algorithm against these rare but mission-critical and safety-crucial scenarios is an important yet challenging task. In this paper, we consider a resilient DRL framework with observational interferences. Under this framework, we discuss the importance of the causal relation and propose a causal inference based DRL algorithm called causal inference Q-network (CIQ). We evaluate the performance of CIQ in several benchmark DRL environments with different types of interferences. Our experimental results show that the proposed CIQ method could achieve higher performance and more resilience against observational interferences.

One-sentence Summary: We propose a causal inference based DRL algorithm called causal inference Q-network (CIQ) under interferences toward resilient learning.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 4 code implementations](https://www.catalyzex.com/paper/arxiv:2102.09677/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=tZb-TuF63