What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization

Xavier Thomas, Deepti Ghadiyaram

Keywords: Representation Learning, Diffusion Models, Domain Generalization

TL;DR: Investigation of Pre-trained feature spaces on how they encode class and domain related structures, and how we can leverage them for domain generalization

Abstract: We investigate how model architectures and pre-training objectives influence feature richness, and introduce a simple method to leverage these features for domain generalization. Given a pre-trained feature space, we first discover latent domain structures, referred to as pseudo-domains, that capture domain-specific variations in an unsupervised manner. We then augment classifiers with these complementary representations, improving generalization to diverse unseen domains. We also analyze how different pre-training feature spaces differ in terms of the granularity of domain-specific variances they capture. Our analysis reveals that diffusion models, in particular, effectively separate domains in their latent spaces. Across five datasets, our approach improves test accuracy by up to 4 over the baseline Empirical Risk Minimization (ERM). Code is available at: https://xthomasbu.github.io/GUIDE/

Submission Number: 25