Disentangling Latent Embeddings with Sparse Linear Concept Subspaces (SLiCS)

Zhi Li, Hau Phan, Matthew Emigh, Austin J. Brockmeier

Keywords: subspace disentanglement, image retrieval, vision-language model, latent embeddings

Abstract: Vision-language co-embedding networks, such as CLIP, provide a latent embedding space with semantic information that is useful for downstream tasks. We hypothesize that the embedding space can be disentangled to separate the information on the content of complex scenes by decomposing the embedding into multiple concept-specific component vectors that lie in different subspaces. We propose a supervised dictionary learning approach to estimate a linear synthesis model consisting of sparse, non-negative combinations of groups of vectors in the dictionary (atoms), whose group-wise activity matches the multi-label information. Each concept-specific component is a non-negative combination of atoms associated to a label. The group-structured dictionary is optimized through a novel alternating optimization with guaranteed convergence. Exploiting the text co-embeddings, we detail how semantically meaningful descriptions can be found based on text embeddings of words best approximated by a concept's group of atoms, and unsupervised dictionary learning can exploit zero-shot classification of training set images using the text embeddings of concept labels to provide instance-wise multi-labels. We apply SLiCS to CLIP embeddings, the highly-compressed autoencoder embeddings from TiTok, and the latent embedding from self-supervised DINOv2. We show that the disentangled embeddings provided by our sparse linear concept subspaces (SLiCS) enable concept-filtered image retrieval that is more precise. In addition to conditional generation using image-to-prompt from the components, we develop a model for conditional generation from each concept subspace using diffusion posterior sampling. Quantitative and qualitative results highlight the improved precision of the concept-filtered image retrieval for all embeddings.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 22775