Fingerprinting the Shadows: Unmasking Malicious Servers with Machine Learning-Powered TLS Analysis

Andreas Theofanous, Eva Papadogiannaki, Alexander Shevtsov, Sotiris Ioannidis

Keywords: TLS, TLS Fingerprinting, Active Probing, Botnet, Command and Control, Server Characterization, Machine Learning, Explainability

Abstract:

Over the last few years, the adoption of encryption in network
traffic has been constantly increasing. The percentage of encrypted
communications worldwide is estimated to exceed 90%. Although
network encryption protocols mainly aim to secure and protect
users’ online activities and communications, they have been exploited
by malicious entities that hide their presence in the network.
It was estimated that in 2022, more than 85% of the malware used
encrypted communication channels.
In this work, we examine state-of-the-art fingerprinting techniques
and extend a machine learning pipeline for effective and practical
server classification. Specifically, we actively contact servers to
initiate communication over the TLS protocol and through exhaustive
requests, we extract communication metadata. We investigate
which features favor an effective classification, while we utilize and
evaluate state-of-the-art approaches. Our extended pipeline can
indicate whether a server is malicious or not with 91% precision and
95% recall, while it can specify the botnet family with 99% precision
and 99% recall.

Track: Security

Submission Guidelines Scope: Yes

Submission Guidelines Blind: Yes

Submission Guidelines Format: Yes

Submission Guidelines Limit: Yes

Submission Guidelines Authorship: Yes

Student Author: Yes

Submission Number: 2531