Transferable and principled efficiency for open-vocabulary segmentation

Jingxuan Xu, Wuyang Chen, Yao Zhao, Yunchao Wei

Abstract:

Recent success of pre-trained foundation vision-language
models makes Open-Vocabulary Segmentation (OVS) possible. Despite the promising performance, this approach
introduces heavy computational overheads for two challenges: 1) large model sizes of the backbone; 2) expensive
costs during the fine-tuning. These challenges hinder this
OVS strategy from being widely applicable and affordable
in real-world scenarios. Although traditional methods such
as model compression and efficient fine-tuning can address
these challenges, they often rely on heuristics. This means
that their solutions cannot be easily transferred and necessitate re-training on different models, which comes at a
cost. In the context of efficient OVS, we target achieving
performance that is comparable to or even better than prior
OVS works based on large vision-language foundation models, by utilizing smaller models that incur lower training
costs. The core strategy is to make our efficiency principled
and thus seamlessly transferable from one OVS framework
to others without further customization. Comprehensive
experiments on diverse OVS benchmarks demonstrate our
superior trade-off between segmentation accuracy and computation costs over previous works