ENS-RFMC: An Encrypted Network Traffic Sampling Method Based on Rule-Based Feature Extraction and Multi-hierarchical Clustering for Intrusion Detection

Liangchen Chen, Shu Gao, Zi-Xuan Wei, Baoxu Liu, Xu-Yao Zhang

Abstract: Efficiently and quickly identifying malware in encrypted network traffic is a major challenge. This research presents and evaluates a novel approach for sampling encrypted network traffic, called ENS-RFMC. Initially, the method employs a rule-based strategy for extracting relevant metadata features of encrypted traffic from network connections, SSL and certificates log files. Subsequently, a multi-hierarchical clustering algorithm is utilized to divide the dataset into smaller clusters, with benign clusters subsequently eliminated for streamline data processing. Classifier models such as Random Forests, XGBoost and SVM are then utilized to detect attacks and assess performance. Experimental results indicate that the proposed ENS-RFMC sampling method is effective, with the encrypted traffic intrusion detection model employing ENS-RFMC sampling exhibits enhanced accuracy and efficiency in identifying attacks.