Understanding Weak-to-Strong Generalization: A Spectral Analysis

ICLR 2026 Conference Submission15001 Authors

Keywords: weak-to-strong generalization, spectral analysis

Abstract: Weak-to-Strong (W2S) generalization, where a student model surpasses its weaker teacher using the teacher's labels, is widely studied recently. We theoretically analyze this problem using a kernel ridgeless regression student in a Reproducing Kernel Hilbert Space (RKHS), learning from a teacher with systematic bias and output variance. Our derived asymptotic bias-variance decomposition reveals how teacher errors are processed by the student. This processing is critically mediated by the student's kernel eigenvalues and, crucially, its choice of operational modes and their alignment with the teacher's signal. We then elucidate precise conditions for W2S: outperformance hinges on this selection effectively managing the trade-off between bias and variance. Such strategic mode utilization can lead to a more favorable bias configuration via selectively ignoring the teacher's biased modes, or a reduction of teacher variance through modes with beneficial eigenvalue properties. Our experiments validate these theoretical conditions, demonstrating successful W2S generalization and underscoring the critical impact of kernel selection on navigating the bias-variance trade-off.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 15001