Distributed response to network intrusions using multiagent reinforcement learning

Kleanthis Malialis, Daniel Kudenko

Abstract: Distributed denial of service (DDoS) attacks constitute a rapidly evolving threat in the current Internet. Multiagent Router Throttling is a novel approach to defend against DDoS attacks where multiple reinforcement learning agents are installed on a set of routers and learn to throttle or rate-limit traffic towards a victim server. It has been demonstrated to perform well against DDoS attacks in small-scale network topologies. The focus of this paper is to tackle the scalability challenge. Scalability is one of the most important aspects of a defence system since a non-scalable defence mechanism will never be considered, let alone adopted, for wide deployment by a company or organisation. In this paper we introduce Coordinated Team Learning (CTL) which is a novel design to the original Multiagent Router Throttling approach. One of the novel characteristics of our approach is that it provides a decentralised coordinated response to the DDoS problem. It incorporates several mechanisms, namely, hierarchical team-based communication, task decomposition and team rewards and its scalability is successfully demonstrated in experiments involving up to 100 reinforcement learning agents. We compare our proposed approach against a baseline and a popular state-of-the-art router throttling technique from the network security literature and we show that our approach significantly outperforms both of them in a series of scenarios with increasingly sophisticated attack dynamics. Furthermore, we show that our approach is more resilient and adaptable than the existing throttling approaches.