Navigating Semantic Drift in Task-Agnostic Class-Incremental Learning

Fangwen Wu, Lechao Cheng, Shengeng Tang, Xiaofeng Zhu, Chaowei Fang, Dingwen Zhang, Meng Wang

Abstract: Class-incremental learning (CIL) seeks to enable a model to sequentially learn new classes while retaining knowledge of previously learned ones. Balancing flexibility and stability remains a significant challenge, particularly when the task ID is unknown. To address this, our study reveals that the gap in feature distribution between novel and existing tasks is primarily driven by differences in mean and covariance moments. Building on this insight, we propose a novel semantic drift calibration method that incorporates mean shift compensation and covariance calibration. Specifically, we calculate each class's mean by averaging its sample embeddings and estimate task shifts using weighted embedding changes based on their proximity to the previous mean, effectively capturing mean shifts for all learned classes with each new task. We also apply Mahalanobis distance constraint for covariance calibration, aligning class-specific embedding covariances between old and current networks to mitigate the covariance shift. Additionally, we integrate a feature-level self-distillation approach to enhance generalization. Comprehensive experiments on commonly used datasets demonstrate the effectiveness of our approach. The source code is available at https://github.com/fwu11/MACIL.git.

Lay Summary:

Modern AI systems typically learn all categories at once, but in real-world scenarios, they must continually absorb new classes over time without forgetting previously acquired knowledge—a challenge known as class-incremental learning (CIL). This becomes particularly difficult when the model lacks task identity information, as internal feature statistics—specifically the mean ("center") and covariance ("spread")—shift in ways that cause representations of older classes to drift.

To address this, we propose a semantic drift calibration method. First, we compensate for mean shifts by tracking how new sample embeddings move relative to existing class centers, capturing task-level shifts as each new class is introduced. Next, we align class-specific covariance using a Mahalanobis distance constraint, effectively calibrating the feature spread between the previous and current models.

Evaluated on standard CIL benchmarks, our method significantly improves accuracy across both old and new classes. By directly correcting the statistical roots of semantic drift, this work moves us closer to AI systems capable of continuous, robust, and graceful adaptation in dynamic environments.

Primary Area: Deep Learning

Keywords: Class-incremental learning, continual learning

Submission Number: 2505