Improved baselines for vision-language pre-training

Enrico Fini, Pietro Astolfi, Adriana Romero-Soriano, Jakob Verbeek, Michal Drozdzal

Event Certifications: iclr.cc/ICLR/2024/Journal_Track

Abstract:

Contrastive learning has emerged as an efficient framework to learn multimodal representations. CLIP, a seminal work in this area, achieved impressive results by training on paired image-text data using the contrastive loss. Recent work claims improvements over CLIP using additional non-contrastive losses inspired from self-supervised learning.
However, it is sometimes hard to disentangle the contribution of these additional losses from other implementation details, \eg, data augmentation or regularization techniques, used to train the model. To shed light on this matter, in this paper, we first propose, implement and evaluate several baselines obtained by combining contrastive learning with recent advances in self-supervised learning.
In particular, we use the loss functions that were proven successful for visual self-supervised learning to align image and text modalities. We find that these baselines outperform a basic implementation of CLIP. However, when a stronger training recipe is employed, the advantage disappears. Indeed, we find that a simple CLIP baseline can also be improved substantially, up to a 25\% relative improvement on downstream zero-shot tasks, by using well-known training techniques that are popular in other subfields. Moreover, we discover that it is enough to apply image and text augmentations to make up for most of the improvement attained by prior works. With our improved training recipe for CLIP, we obtain state-of-the-art performance on four standard datasets, and consistently outperform prior work (up to +4\% on the largest dataset), while being substantially simpler.

Certifications: Featured Certification

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

- Added one experiment on a larger dataset, PMD46M (see Table 3)
- Added one experiment using a larger vision backbone, ViT-L/16 (see Figure 4 + part of Table 4)
- Added an ablation of the $\alpha$ and $\beta$ hyper-parameters, see supplementary (Table 6)
- Fixed typos

Code: https://github.com/facebookresearch/clip-rocket

Assigned Action Editor: ~Hanwang_Zhang3

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1158