Intelligent Security Aware Routing: Using Model-Free Reinforcement Learning

Anand Mudgerikar, Elisa Bertino

Abstract: With the emergence and successful deployment of software defined networks (SDN), zero-trust security architecture, and network function virtualization (NFV) in large scale modern enterprise and 5G networks, it is possible to build ‘smart’ network controllers that leverage machine learning (ML) to learn policies for optimal and secure traffic engineering. Deep Reinforcement Learning (DRL) is an effective technique for building such smart controllers because of its model-free nature and ability to learn policies dynamically through experience without requiring extensive training data. However, conventional DRL frameworks are geared to maximize functionality and do not take network security into account. To address such a gap, we propose a security-aware DRL framework, STE-SDN that learns ‘intelligent policies' for traffic engineering (routing) to both maximize functionality gain and minimize security risk. We instantiate our framework in a simulated SDN environment of 5000 nodes with different security services and three attack classes: DDoS, Web-based and Brute-Force attacks. We then analyze our framework using the CICIDS-17 dataset in terms of performance and effectiveness in mitigating security risks. We find that our RL framework reduces detection loss by 85.8% and maintains close to optimal performance for 78.8%.