DQN-MSRA: an online SFC deployment method based on multistep reinforcement learning

Zhibo Zhang, Huiqiang Wang, Rongqiang Li, Hongwu Lv, Dongmiao He

Abstract: Nowadays, with enhancements in software-defined network (SDN) technology and network function virtualization technology, the IETF has proposed replacing conventional dedicated hardware-based network services with virtualized service function chains (SFCs). However, in dynamically changing network environments, performing online SFC deployment while satisfying network capacity constraints is already a challenging task. Providing fast and efficient online services for randomly arriving requests while also ensuring service quality is even more difficult. In this regard, we incorporate the SDN characteristic and propose a univariate modeling approach that can accommodate both network function placement and dynamic routing. This pattern not only avoid exploiting invalid paths during the training procedure, but also effectively reduces the number of variables in the complete SFC placement, as well as the size of the action space. More importantly, we apply the idea of multistep reinforcement learning (RL) to SFC placement with dependencies for the first time and propose a novel SFC-suitable placement method named DQN-MSRA. Compared with the conventional one-step RL process that focuses on the impact of the current step reward on a given action, DQN-MSRA reduces the impact of the immediate reward and pays more attention to the long-term ones, making it more applicable to online SFC placement. We verify the effectiveness of DQN-MSRA from multiple perspectives and experimentally analyze the association between the aggregation window size and service chain length. In addition, our approach has a higher acceptance rate, better deployment stability, and higher adaptability than recent novel deployment methods.