Nature vs. Nurture: Limits of Representation Universality Reveal Architectural and Training Differences in Image Models

Fredo Guan, Shreyan Banerjee, Aditya Bajoria, Akarsh Shetty, Cole Blondin

Track: long paper (up to 10 pages)

Domain: machine learning

Abstract: Recent works have hypothesized and demonstrated the general inter-compatibility and translatability of vector embeddings in learned embedding spaces. We study the limits of representation compatibility across image models encompassing a variety of architectures and training paradigms. We train adapters, comprised of linear encoders and decoders, to translate between all pairs of embedding spaces using a shared latent space. We then evaluate cross-model and cross-domain compatibility using classification, retrieval, and embedding-based benchmarks, revealing limits in compatibility across learned embedding spaces. Further analysis of pairwise translation performance reveals phylogenetic patterns that reflect the fundamental differences in model architecture and training.

Presenter: ~Fredo_Guan1

Submission Number: 49