Anti-EMP: Encrypted Malware Packets Filtering Algorithm Leveraging Ciphertext Patterns Under Zero Knowledge Setting

Junggab Son, Jeehyung Kim, Jemin Ahn, Doowon Kim, Homook Cho, Daeyoung Kim

Abstract: Malware often employs encryption to obfuscate its network communications, posing challenges for network-based anomaly detection techniques. Distinguishing encrypted packets from one another becomes particularly difficult, especially when operating under zero knowledge setting, such as detecting new malware. Existing approaches rely on unique features extracted from network connections to train deep learning algorithms. However, these methods fall short when dealing with new malware due to limited information. To address this challenge, we propose Anti-EMP, an algorithm designed to filter encrypted malware packets. Specifically, Anti-EMP identifies and sifts out encrypted packets originating from the same malware across multiple clients. Our approach is grounded in two practical assumptions: (a) the packets were encrypted using an unknown, identical stream cipher and encryption key and (b) a suspicious packet related to malware can be captured. We also propose a novel method for generating Anti-EMP, significantly enhancing the capability to detect encrypted malware packets without prior knowledge, i.e., zero knowledge settings. Our experiments show that Anti-EMP can be generated in approximately one second, facilitating easy iteration and easy selection of another suspicious packet if it turns out to be ineffective. Notably, our proposed scheme demonstrates high effectiveness, achieving a True-Positive Rate (TPR) of 0.998 and a False-Positive Rate (FPR) of 0.001.

External IDs:dblp:conf/securecomm/SonKAKCK24