Byzantine-robust federated reinforcement learning via critical parameter analysis

Wenzheng Jiang, Ji Wang, Weidong Bao, Yaohong Zhang

Abstract: Federated Reinforcement Learning (FRL) has emerged as a promising approach to facilitate collaborative policy learning among distributed agents while preserving data privacy. However, the susceptibility of FRL to Byzantine attacks undermines its performance and exacerbates uncertainties during the training process. In practical deployment, effectively mitigating Byzantine attacks instigated by malicious agents poses a formidable challenge. To confront this issue, we present a robust FRL framework, Fed-DQC, which integrates a novel Byzantine-resilient mechanism into Double Deep Q-Network (DDQN). The core of Fed-DQC lies in its distinctive parameter analysis and model similarity measurement mechanism. This mechanism can effectively identify and mitigate the influence of malicious agents, thereby curtailing the destructive impact of Byzantine attacks on the training process. Leveraging the distinctive nature of reinforcement learning, we propose an innovative two-stage federated aggregation scheme designed to maximize the utilization of agents’ exploration processes while minimizing the risk of misidentifying malicious agents. Our extensive experiments show that Fed-DQC exhibits strong robustness and adaptability against five different Byzantine attacks, achieving even better performance than scenarios unaffected by Byzantine attacks.

External IDs:dblp:journals/mlc/JiangWBZ25