SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP

Christoph Timmermann, Hyunse Lee, Woojin Lee

Keywords: Multi-modal learning, few-shot classification, efficient adaptation, modality gap, intra-modal misalignment, CLIP, vision-language model

TL;DR: SeMoBridge is a lightweight method that addresses CLIP's unreliable image-to-image comparisons by transforming images to the text modality, achieving superior few-shot accuracy in low-data scenarios with minimal training time.

Abstract:

While Contrastive Language-Image Pretraining (CLIP) excels at zero-shot tasks by aligning image and text embeddings, its performance in few-shot classification is hindered by a critical limitation: intra-modal misalignment. This issue, caused by a persistent modality gap and CLIP's exclusively inter-modal training objective, leaves the embedding spaces uncalibrated, making direct image-to-image comparisons unreliable. Existing methods attempt to address this by refining similarity logits or by computationally expensive per-sample optimization.
To overcome these challenges, we introduce SeMoBridge, a lightweight yet powerful approach that directly addresses the misalignment. Our method maps images into the text modality, while keeping their semantic content intact through what we call a Semantic Modality Bridge. SeMoBridge is closed-form and can optionally be trained through multi-modal supervision, combining image and text-alignment losses to optimize the projection. Experiments show that the trained version, SeMoBridge-T, requires only a fraction of the training time while overall outperforming other methods, particularly in low-data scenarios (1, 2, and 4 shots).

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 8650