Byzantine Resilient Federated REINFORCE (GM-FedREINFORCE)

Ankit Pratap Singh, Ronak Tali

Abstract: Recent advancements in federated learning have inspired the emergence of federated reinforcement learning, wherein multiple agents learn a policy in a federated manner without sharing trajectories. However, one of the key challenges in federated learning lies in dealing with adversarial attacks, such as Byzantine attacks. This work addresses this challenge by introducing the GM-FedREINFORCE algorithm, which extends the REINFORCE method-an established policy gradient technique-and provides robust theoretical guarantees on its convergence even in the face of adversarial attacks. We do this by replacing the mean step of GD by Geometric median. Remarkably, our algorithm remains effective when less than half of the agents are under attack, ensuring robust learning. To validate our theoretical claims, we conducted empirical experiments on benchmark RL environments with both discrete and continuous action spaces. Specifically, we employed the Cart-Pole environment to evaluate discrete action spaces and the Cart-Pole Swing-Up environment to assess continuous action spaces. Our experimental results confirm the effectiveness of our algorithm, showcasing its ability to successfully learn policies even under Byzantine attacks.

External IDs:dblp:conf/icmla/SinghT23