TraMEL: An Exemplar Replay-Based Continual Learning Framework for Malware Traffic Analysis

ICLR 2026 Conference Submission20123 Authors

Keywords: Malware Detection, Continual Learning, Machine Learning

Abstract:

Most prior work on continual malware detection has centered on static code analysis. In contrast, this paper explores continual learning (CL) for malware traffic analysis, which leverages encrypted flow features to capture behavioral signals resilient to obfuscation and encryption. Unlike intrusion detection systems that focus on coarse anomaly detection, malware traffic analysis requires fine-grained family-level classification under evolving and imbalanced distributions, making it a distinct and challenging setting for CL.

We introduce TraMEL (Traffic-based Malware Exemplar Learning), a replay-based CL framework designed for malware traffic. TraMEL integrates (i) adaptive exemplar selection to balance long-tailed family distributions and (ii) an exemplar refinement phase to counter task recency bias while operating under strict memory budgets. We evaluate both standard class-incremental and temporally shifted scenarios, showing that TraMEL consistently outperforms strong CL baselines such as iCaRL, ER, and TAMiL by 10–30 percentage points on CICAndMal2017 and IoT23. Remarkably, it approaches the performance of joint training, a theoretical upper bound with full access to past data. These results demonstrate that CL on malware traffic is both feasible and practical, providing a memory-efficient approach for real-world detection. Code is available at \url{ https://anonymous.4open.science/r/ICLR2026-code-D575/}.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 20123