Reverse Distillation: Disentangling and Scaling Protein Language Model Representations

ICLR 2026 Conference Submission22152 Authors

Keywords: Protein language models, model scaling, Representation learning, Subspace decomposition, interpretability, Model distillation

TL;DR: Protein language models plateau. Reverse Distillation decomposes them via smaller models, improving scalability.

Abstract: Unlike the foundation model scaling laws seen in natural language processing and computer vision, biological foundation models scale relatively poorly. For example, the ESM-2 family of protein language models plateaus at 650M-3B parameters on ProteinGym benchmarks. We address this limitation by introducing Reverse Distillation, a principled framework that decomposes large protein language model representations into orthogonal subspaces guided by smaller models of the same family. We hypothesize that this decomposition matches the natural hierarchy of protein properties, where broad features like secondary structure are robustly captured by compact, smaller models while the residual capacity of larger models specializes in protein-family specific functions. Our method is theoretically grounded and enables monotonic scaling---larger reverse-distilled models consistently outperform their smaller counterparts, overcoming the scaling plateau. Moreover, on ProteinGym benchmarks, reverse-distilled ESM-2 variants broadly outperform their respective baseline models at the same embedding dimensionality. Our approach offers a generalizable framework for disentangling hierarchical feature spaces in foundation model embeddings, with potential applications across biology and other domains where scaling challenges persist.

Primary Area: applications to physical sciences (physics, chemistry, biology, etc.)

Submission Number: 22152