Dynamic Rank Adjustment for Accurate and Efficient Neural Network Training

Hyuntak Shin, Aecheon Jung, Sungeun Hong, Sunwoo Lee

Keywords: Low-rank training, Rank adjustment, model reparameterization

TL;DR: Mitigating rank decline effect in low-rank training via dynamic rank adjustment.

Abstract: Low-rank training is a primary strategy for efficient deep learning, but it presents a fundamental challenge. It reduces computational cost, yet it permanently caps a model’s representational capacity and accelerates the rank collapse that diminishes its expressive power during training. We address this with dynamic-rank training, a framework built on the intuition that a model can temporarily escape its low-rank constraints to restore its full learning potential. Our approach strategically interleaves full-rank epochs within a low-rank schedule, with the timing of these restorative phases aligned with the learning rate’s noise regimes to maximize their effect. This enables the model to regain expressive power at critical stages of training by restoring the effective rank of its weights. Our extensive evaluations across various computer vision and natural language processing benchmarks show this method achieves the accuracy of full-rank models while retaining the computational advantages of low-rank training.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 10681