Towards Environmental Robustness in Deep Reinforcement Learning

Chenxu Wang, Huaping Liu

Keywords: Deep Reinforcement Learning; Adversarial Perturbation; Adversarial robustness;

TL;DR: We propose a new threat model in which the attackers can only perturb the environmental states and design an attack method and a defense method.

Abstract:

Following the widespread application of Deep Reinforcement Learning (DRL) in robotics and other domains, adversarial attacks and robustness in DRL have also been widely studied in various threat models. However, most of them assume runtime access of the victim, which limits the feasibility of the attacks.
To evaluate the robustness more practically, in this paper, we propose a threat model in which the attacker can only inflict static environmental perturbations on the initial state. By designing a preliminary non-targeted attack method and performing a case study on policy-based DRL agents, we show that the agents are still assailable in our threat model even though the capability of attackers has been severely limited due to the feasibility consideration.
We also propose a defense framework, named Boosted Adversarial Training (BAT), which incorporates a supervised kick-starting stage before adversarial training to avoid failure. Extensive experimental results demonstrate that our BAT framework can significantly enhance the robustness of agents in all situations while the existing robust reinforcement learning algorithms may not be suitable.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1363