Malicious encrypted traffic features extraction model based on unsupervised feature adaptive learning

Zhihong Zhou, Hu Bin, Jianhua Li, Ying Yin, Xiuzhen Chen, Jin Ma, Lihong Yao

Abstract: As network traffic is increasingly valued for privacy protection and the encrypted SSL/TLS (Secure Sockets Layer/Transport Layer Security) traffic is surging, more and more malicious behaviors are hidden in it. Current detection methods are less accurate in detecting new and unknown malicious traffic. Although the method based on the supervised machine learning model has excellent accuracy performance, it has low detection strength and poor scalability for new and unknown malicious traffic. Therefore, this paper proposes a malicious SSL/TLS traffic detection method based on feature adaptive learning. The model can automatically learn key classification information from the unmarked malicious SSL/TLS encrypted traffic, and uses the 5-Tuple-Masking technology to optimize the input data, which greatly enhances the model's adaptation ability to new malicious traffic in complex network environments. After experimental verification, its comprehensive accuracy rate reaches 89.25%. Moreover, the supervised convolutional neural network detection method is used to compare and test the feasibility of this model in the field of malicious SSL/TLS traffic detection.