Reward-Free Action Poisoning in Offline RL via Conditional Shapley Value Estimation

Shenghong He, Chao Yu, Kaiqiang Ke, Yinqi Wei

Keywords: Poisoning Attack, Offline reinforcement learning, Data Security, Reward-free Attack

TL;DR: SAPA is a reward-free poisoning attack for offline RL that uses Shapley values to target key actions, theoretically guarantees policy degradation, and empirically outperforms prior methods.

Abstract: Recent studies show that data poisoning attacks can degrade the performance of offline reinforcement learning (RL) policies by strategically tampering with training datasets. However, existing methods generally assume the acquisition of reward signals during the generation of poisoning data, thus limiting their applications in real-world scenarios when the reward signals are not available. In this paper, we propose a novel method, called Shapley Action-Poisoning Attack (SAPA), which calculates the contribution of each state-action pair in a trajectory to identify key actions for poisoning attacks without dependence on reward signals. Theoretical analysis proves that SAPA can degrade the performance of the learned policy below a specified threshold by tampering with key actions. Numerous experimental results demonstrate that SAPA surpasses state-of-the-art poisoning methods in the attack performance under various offline algorithms.

Area: Learning and Adaptation (LEARN)

Generative A I: I acknowledge that I have read and will follow this policy.

Submission Number: 912