An Automatic XSS Attack Vector Generation Method Based on the Improved Dueling DDQN Algorithm

Yuan Yao, Junjiang He, Tao Li, Yunpeng Wang, Xiaolong Lan, Yuan Li

Abstract: Traditional XSS (Cross Site Scripting) scanners typically rely on attack vectors based on expert knowledge and manual testing, which not only incur high costs and long processing times but also result in a significant number of false positives. In this paper, we build an automatic XSS attack vector generation method based on the improved Dueling DDQN algorithm. First, we model the XSS attack vector generation process as a Markov decision process, mapping the initial attack vector mutation points and mutation strategies to the state space and action space of the model, respectively. Second, we propose an improved Dueling DDQN algorithm by introducing a priority experience replay mechanism to improve algorithm performance and the speed of attack vector generation. Third, we establish a feedback mechanism based on the edit distance algorithm to define the role of the reward function, preventing the model from getting stuck in local optima and achieving better mutation effects. Finally, we propose an automatic XSS attack verification method based on static semantic analysis to validate the effectiveness of our generated attack vectors. Based on the mentioned approach, we have developed a prototype tool for automatic XSS scanning. The experimental results demonstrate that the improved Dueling DDQN algorithm outperforms other value-based reinforcement learning algorithms in terms of convergence speed, learning efficiency, and stability. The adaptive attack vector generation model can generate attack vectors that adapt to program context semantics and bypass defense mechanisms.