Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping

Jinwoo Baek

Keywords: spectral conditioning, Chebyshev moments, condition number, regularization, deep learning optimization, training stability

TL;DR: We propose Chebyshev Moment Regularization (CMR), a new loss function that directly controls the layer spectrum of deep learning models to dramatically improve the condition number and restore unstable training.

Abstract: We introduce Chebyshev Moment Regularization (CMR), a simple, architecture-agnostic loss that directly optimizes layer spectra. CMR jointly controls spectral edges via a log-condition proxy and shapes the interior via Chebyshev moments, with a decoupled, capped mixing rule that preserves task gradients. We prove strictly monotone descent for the condition proxy, bounded moment gradients, and orthogonal invariance. In an adversarial $\kappa$-stress setting (MNIST, 15-layer MLP), compared to vanilla training, CMR reduces mean layer condition numbers by $\sim10^3$ (from $\approx3.9\times10^3$ to $\approx3.4$ in 5 epochs), increases average gradient magnitude, and restores test accuracy ( $\approx10\to\approx86$ ). These results support optimization-driven spectral preconditioning: directly steering models toward well-conditioned regimes for stable, accurate learning.

Submission Number: 164