Bridging Generative and Predictive Paradigms via Hidden-Self-Distillation

Scott C. Lowe, Anthony Fuller, Sageev Oore, Evan Shelhamer, Graham W. Taylor

Track: tiny paper (up to 4 pages)

Keywords: JEPA, MAE, SSL, self-distillation

TL;DR: Predicting multiple layers of abstraction improves SSL performance

Abstract: The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g., MAE) that reconstruct raw low-level data, and predictive approaches (e.g., I-JEPA) that predict high-level abstract embeddings. While generative methods provide strong grounding, they are computationally inefficient for high-redundancy modalities like vision, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer from training instability due to their reliance on final-layer self-distillation. We introduce Bootleg, a method that bridges this divide by tasking the model with predicting continuous latent representations from multiple hidden layers of a teacher network. This hierarchical objective forces the model to capture features at varying levels of abstraction simultaneously. We demonstrate that Bootleg significantly outperforms comparable baselines on classification of ImageNet-1K and iNaturalist, and ADE20K segmentation (+10% over I-JEPA). This positions Bootleg as an ideal "representation interface" for next-generation multimodal models, where the optimal training signal lies neither at the pixel level nor the semantic peak, but in the rich intermediate hierarchy.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 72