COLA: Compressed Latent Adapter Memory for Lifelong Learning

Keywords: Continual learning, Lora

Abstract:

Continual learning (CL) for large language models remains challenging due to catastrophic forgetting and the prohibitive cost of retraining large models as new tasks arrive. Existing approaches either rely on rehearsal buffers, store task-specific parameters that grow linearly with tasks, or restrict model plasticity. We propose COLA (Compressed Latent Adapter Memory), a rehearsal-free continual learning framework that stores task knowledge in a compact latent space rather than explicit model parameters. COLA trains lightweight LoRA adapters for each task and compresses the adapter weights using a contractive autoencoder into low-dimensional latent codes. For task-agnostic inference, COLA retrieves the most relevant latent adapter code using input-conditioned task keys and loads the reconstructed adapter into the frozen backbone. Across intent classification and task-oriented
dialogue benchmarks, COLA outperforms strong rehearsal-free baselines, approaches joint multi-task training performance, and substantially reduces per-task storage. These results show that latent adapter memory is an effective mechanism for rehearsal-free continual learning in language models.

Submission Number: 31