LR-Adapter: Low-Rank Tensor Adapters for Vision Foundation Models

Keywords: parameter-efficient fine-tuning, tensor decomposition, test-time adaptation, vision transformers, low-rank adapters

Abstract:

Adapting vision foundation models to downstream tasks under annotation
scarcity and distribution shift remains challenging.
We propose \textbf{LR-Adapter}, a family of residual adapters whose internal
weight tensors are structured as low-rank Canonical Polyadic Decomposition
(CPD) or Tucker factors from the outset.
Compared with a full-rank bottleneck adapter, LR-Adapter reduces trainable
parameters by up to $83\%$ while matching or exceeding accuracy across
14 vision datasets with a frozen DINOv2
ViT-S/14 backbone.
At test time, updating only the compact factor matrices via entropy
minimization provides stable domain adaptation without backbone access,
yielding lower Expected Calibration Error than full-rank counterparts.

Submission Number: 24