Go to ICLR 2026 Conference homepageBreaking the Limits of Open-Weight CLIP: An Optimization Framework for Self-supervised Fine-tuning of CLIP
Keywords: self-supervised learning, CLIP, optimization, fine-tuning, contrastive loss
TL;DR: We introduce TuneCLIP, a novel self-supervised optimization framework designed to enhance state-of-the-art pretrained open-weight CLIP models.
Abstract: CLIP has become a cornerstone of multimodal representation learning, yet improving its performance typically requires a prohibitively costly process of training from scratch on billions of samples. We ask a different question: *Can we improve the performance of open-weight CLIP models across various downstream tasks using only existing self-supervised datasets?* Unlike supervised fine-tuning, which adapts a pretrained model to a single downstream task, our setting seeks to improve general performance across various tasks. However, as both our experiments and prior studies reveal, simply applying standard training protocols starting from an open-weight CLIP model often fails, leading to performance degradation. In this paper, we introduce **TuneCLIP**, a self-supervised fine-tuning framework that overcomes the performance degradation. TuneCLIP has two key components: (1) a warm-up stage of recovering optimization statistics to reduce cold-start bias, inspired by theoretical analysis, and (2) a fine-tuning stage of optimizing a new contrastive loss to mitigate the penalization on false negative pairs. Our extensive experiments show that TuneCLIP consistently improves performance across model architectures and scales. Notably, it elevates leading open-weight models like SigLIP (ViT-B/16), achieving gains of up to +2.5\% on ImageNet and related out-of-distribution benchmarks, and +1.2\% on the highly competitive DataComp benchmark, setting a new strong baseline for efficient post-pretraining adaptation.
Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Submission Number: 21144
Loading