Adaptive Log Anomaly Detection through Data–Centric Drift Characterization and Policy-Driven Lifelong Learning

Keywords: Log Anomaly Detection, Concept Drift, Lifelong Learning, Semantic Drift, Syntactic Drift, HDFS, Apache, BGL, ADWIN

Abstract:

Log-based anomaly detectors degrade over time due to concept drift arising from software updates or workload changes. Existing systems typically react by retraining entire models, leading to catastrophic forgetting and inefficiencies. We propose3
an adaptive framework that first classifies drift in log data into semantic (frequency shifts within known templates) and syntactic (emergence of new log templates) categories via statistical tests and novelty detection. Based on the identified drift type, a policy-driven lifelong learning manager applies targeted updates—experience replay to mitigate forgetting under semantic drift and dynamic model expansion to accommodate syntactic drift. This approach is validated on semi-synthetic logs and real-world longitudinal datasets (HDFS, Apache, and BGL), maintaining high F1-scores, reducing computational overhead, and preserving historical knowledge compared to monolithic retraining.

Submission Number: 117