TwinTURBO: Semi-Supervised Fine-Tuning of Foundation Models via Mutual Information Decompositions for Downstream Task and Latent Spaces

Guillaume Quétant, Pavlo Molchanov, Slava Voloshynovskiy

Keywords: Deep Learning, Machine Learning, Information Theory, Mutual Information, Bounds, Foundation Model, Fine-Tuning, Classification, Semi-Supervised, Unsupervised, Contrastive

TL;DR: We propose a semi-supervised fine-tuning framework for foundation models using two mutual information bounds, achieving significant classification improvements with minimal labelled data.

Abstract:

We present a semi-supervised fine-tuning framework for foundation models that utilises mutual information decomposition to address the challenges of training for a limited amount of labelled data.
Our approach derives two distinct lower bounds: i) for the downstream task space, such as classification, optimised using conditional and marginal cross-entropy alongside Kullback-Leibler divergence, and ii) for the latent space representation, regularised and aligned using a contrastive-like decomposition.
This fine-tuning strategy retains the pre-trained structure of the foundation model, modifying only a specialised projector module comprising a small transformer and a token aggregation technique.
Experiments on several datasets demonstrate significant improvements in classification tasks under extremely low-labelled conditions by effectively leveraging unlabelled data.

Submission Number: 57