Retain and Adapt: Auto-Balanced Model Editing for Open-Vocabulary Object Detection under Domain Shifts

Zixuan Duan, Fengyuan Lu, Xunzhi Xiang, Wenbin Li, Yang Gao, Qi Fan

Keywords: Open-Vocabulary Object Detection, Model Editing, Continual Learning, Knowledge Injection, Few-Shot Learning, Catastrophic Forgetting

TL;DR: We propose a hyperparameter-free auto-balanced model editing method that flexibly injects and learns new task knowledge into open-vocabulary detectors while preserving original capabilities, achieving strong adaptation without retraining.

Abstract:

Recent advances in Open Vocabulary Object Detection (OVOD) have shown strong performance on standard benchmarks, but performance drops sharply under out-of-distribution (OOD) shifts. Continual learning offers a potential remedy by sequentially integrating new tasks, yet existing methods often struggle to balance retaining the pre-trained model capabilities with adapting to new tasks, and usually require retraining under specific task orders. To address these limitations, we observe that model editing naturally lends itself to this setting, as it enables efficient knowledge injection while retaining prior capabilities. Building on this insight, we introduce $\textbf{A}$utomatically $\textbf{B}$alanced $\textbf{M}$odel $\textbf{E}$diting ($\textbf{ABME}$), which injects new task knowledge into the powerful OVOD models while preserving the model’s original abilities. We first stores compact key–value representations with storage cost independent of task volume. Then we leverage the stored KV matrices to automatically balance the new and old knowledge for varying learning scenarios,
supporting order-agnostic task insertion or removal without additional retraining. Experiments show that ABME consistently achieves a better trade-off between maintaining pre-trained performance and adapting to diverse OOD tasks compared to existing continual learning approaches for open-vocabulary object detection, and generalizes seamlessly across different models and task scales.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 5519