High Fidelity Visualization of What Your Self-Supervised Representation Knows About

Florian Bordes, Randall Balestriero, Pascal Vincent

Keywords: self-supervised learning, visualization, diffusion model, conditional generative model, representation

Abstract: Discovering what is learned by neural networks remains a challenge. In self-supervised learning, classification is the most common task used to evaluate how good a representation is. However, relying only on such downstream task can limit our understanding of how much information is contained in the representation of a given input. In this work, we study how to visualize representations learned with self-supervised models. We investigate a simple gradient descend based method to match a target representation and show the limitations of such techniques. We overcome these limitations by developing a representation-conditioned diffusion model (RCDM) that is able to generate high-quality inputs that share commonalities with a given representation. We further demonstrate how our model's generation quality is on par with state-of-the-art generative models and how the representation conditioning brings new avenues to analyze and improve self-supervised models.

One-sentence Summary: We introduce a high fidelity visualization method to get insights about what information is contained in self-supervised representations.