Escaping Label Subspaces via Label Geometry

Jitian Zhao, Nicholas Roberts, Xintong Li, Frederic Sala, Dyah Adila, Sonia Cromp, Tzu-Heng Huang

Abstract: We propose a simple approach for learning in label spaces of extremely high cardinality. In this setting, only a subset of the labels are observed at training time, but we have access to metric information that relates the labels---a common scenario in zero-shot learning, hierarchical classification, and structured prediction. Our technique adapts trained models to produce predictions of unobserved classes. We provide three theoretical insights. First, we give a characterization of the scenarios in which it is possible to predict any unobserved class. Next, we introduce an optimal active learning-like next class selection procedure for when it is not possible to do so. Lastly, we study learning-theoretic tradeoffs between label space richness, sample complexity, and model dimension. Empirical results show that it is possible to use our approach to gain up to 19.5% improvement on pre-trained zero-shot models like CLIP.