TraMEL: An Exemplar Replay-Based Continual Learning Framework for Malware Traffic Analysis

Jungmin Park, Jimin Park, AHyun Ji, Mohammad Saidur Rahman, Se Eun Oh

Keywords: Malware Detection, Continual Learning, Machine Learning

Abstract:

Most prior work on continual malware detection has focused on static code analysis. In contrast, this paper explores continual learning (CL) for malware traffic analysis (MTA), which leverages encrypted flow features to capture behavioral signals that remain observable despite obfuscation and encryption. Unlike conventional intrusion detection systems that perform coarse anomaly detection, MTA requires fine-grained family-level classification under evolving, imbalanced, and non-stationary distributions, making it a distinct and challenging setting for CL.

We introduce TraMEL (Traffic-based Malware Exemplar Learning), a replay-based CL framework designed for MTA. TraMEL integrates (i) adaptive exemplar selection to address long-tailed family distributions and (ii) an exemplar refinement phase to mitigate task recency bias under strict memory budgets. We evaluate TraMEL under both standard class-incremental and temporally shifted scenarios. Across CICAndMal2017 and IoT23, TraMEL outperforms strong CL baselines including iCaRL, ER, and TAMiL by 10–30 percentage points, and approaches the performance of joint training, a theoretical upper bound with full access to past data. These results demonstrate that CL on malware traffic is both feasible and practical, providing a memory-efficient approach toward real-world malware detection. Code is available at \url{https://anonymous.4open.science/r/ICLR2026-code-D575/}.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 20123