Interference-Aware Adapter Routing for Continual Vision–Language Models

Canran Xiao, Zhang Changyuan, Liwei Hou

Keywords: VLM, LoRA, Continual learning

Abstract: Continual adaptation of vision–language models (VLMs) with parameter-efficient modules often suffers from cross-task interference, causing negative transfer and forgetting. We propose AIR, an interference-aware adapter routing framework. AIR estimates conflict online via low-cost gradient surrogates, routes each sample to least-interfering experts, and expands capacity only along high-conflict directions through adaptive LoRA rank; a subspace-packing regularizer enlarges principal angles between experts. A simple analysis links forgetting to these angles and gradient conflict, motivating AIR’s design. Across open-vocabulary classification, image–text retrieval, and multimodal QA, AIR increases average accuracy and reduces forgetting versus replay, distillation, and MoE/parallel-LoRA baselines—while preserving zero-shot retention and cross-modal geometry (lower alignment-isometry error) at comparable or lower memory/compute.

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: true

Submission Guidelines: true

Anonymous Url: true

No Acknowledgement Section: true

Submission Number: 9501