Riding the Gradient: Stable Differentiable SVD and Its Application to LLM Compression

Keywords: SVD, singular value decomposition, differentiable SVD, LLM compression

TL;DR: We propose a stable differentiable SVD that improves low-rank LLM compression in both quality and runtime

Abstract: Singular value decomposition (SVD) underlies many machine learning methods, but differentiating through the full decomposition is unstable in degenerate settings. We develop a stable differentiable SVD by computing it via the polar decomposition and deriving a stable backward formula for the polar factors, which in turn yields a fully differentiable SVD. We further show that the resulting method improves low-rank compression of large language models by enabling better layerwise rank selection and outperforming prior SVD-based pipelines in both quality and runtime.

Submission Number: 99