Measuring the Measures: Discriminative Capacity of Representational Similarity Metrics Across Model Families

Jialin Wu, Shreya Saha, Yiqing Bo, Meenakshi Khosla

Track: Extended Abstract Track

Keywords: Representation Alignment, Representational Similarity, Model Separation, Transformer, CNN

Abstract: Representational similarity metrics are fundamental tools in neuroscience and AI, yet we lack systematic comparisons of their discriminative power across model families. We introduce a quantitative framework to evaluate representational similarity measures based on their ability to separate model families—across architectures (CNNs, Vision Transformers, Swin Transformers, ConvNeXt) and training regimes (supervised vs. self-supervised). Using three complementary separability measures—d-prime from signal detection theory, silhouette coefficients and ROC-AUC—we systematically assess the discriminative capacity of commonly used metrics including RSA, linear predictivity, Procrustes, and soft matching. We show that separability systematically increases as metrics impose more stringent alignment constraints. Among mapping-based approaches, soft-matching achieves the highest separability, followed by Procrustes alignment and linear predictivity. Non-fitting methods such as RSA also yield strong separability across families. These results provide the first systematic comparison of similarity metrics through a separability lens, clarifying their relative sensitivity and guiding metric choice for large-scale model and brain comparisons.

Submission Number: 140