Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning

Arghamitra Talukder, Philippe Chlenski, Itsik Pe'er

Keywords: Privileged basis, Matryoshka representation learning, symmetry breaking, nested representations, PCA, LDA, subspace recovery

TL;DR: Matryoshka representation learning doesn't just order latent dimensions — it pins them to specific, objective-aligned axes, which we prove recovers PCA and LDA exactly in the linear case.

Abstract: Learned representations are often invariant to rotational transformations, leaving individual dimensions non-identifiable and interchangeable. We study how Matryoshka Representation Learning (MRL) induces a task-aligned privileged basis distinct from variance-based or regularizer induced orderings. In the linear setting, we prove that full-prefix MRL recovers the ordered principal directions, and can be computed efficiently using shared statistics. Empirically, we demonstrate that MRL yields consistent per-dimension structure aligned with task signal, where coordinate magnitude reflects informativeness.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 18