Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing

Meng Lou, Stanley Yu, Yizhou Yu

Keywords: Vision Models, Parameter-efficient Fine-tuning, Dynamic Parameter Routing

TL;DR: A new adapter-style PEFT method featuring a mixture-of-experts architecture.

Abstract:

Adapting vision models using parameter-efficient fine-tuning (PEFT) remains challenging, as it aims to achieve performance comparable to full fine-tuning using a minimal number of trainable parameters. When applied to complex dense prediction tasks, existing methods exhibit limitations, including input-agnostic modeling and redundant cross-layer representations. To address these limitations, we propose AdaRoute, a new adapter-style method featuring a simple mixture-of-experts (MoE) architecture. Specifically, we introduce shared expert centers, where each expert is a trainable parameter matrix. During a feedforward pass, each AdaRoute module in the network dynamically generates weight matrices tailored for the current module via a simple dynamic parameter routing mechanism, which selectively aggregates parameter matrices in the corresponding expert center. Dynamic weight matrices in AdaRoute modules facilitate low-rank adaptation in an input-dependent manner, thus generating more customized and powerful feature representations.
Moreover, since AdaRoute modules across multiple network layers share the same expert center, they improve feature diversity by promoting implicit cross-layer feature interaction. Extensive experiments on diverse vision tasks demonstrate the superiority of AdaRoute. For instance, in the object detection and instance segmentation task on COCO2017 with ConvNeXt-L, AdaRoute significantly exceeds full fine-tuning by 1.4\%/1.6\% in AP$^b$/AP$^m$ using less than 5\% of the trainable parameters. In the more challenging panoptic segmentation task, when Swin-B and ConvNeXt-B are used as the backbone, AdaRoute remarkably improves over AdaptFormer by 1.7\% and 2.0\% in PQ, respectively, while using a comparable number of trainable parameters.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 1968