ALRA: Adaptive Low-Rank Approximations for Neural Network Pruning

Soumen Sinha, Rajen Kumar Sinha

Abstract: In the realm of deep learning, the escalating complexity of neural networks has posed substantial computational challenges. This paper introduces a novel solution, Adaptive Low-Rank Approximations (ALRA), which marks a paradigm shift in model optimization. ALRA's distinctiveness lies in its adaptive nature; unlike conventional fixed-rank approximations, it dynamically adjusts the rank of factorized weight matrices during model training through Singular Value Decomposition (SVD). This adaptive capability enables deep learning models to autonomously tailor their complexity to the inherent data distribution, leading to a reduction in the number of model parameters and an improvement in predictive accuracy. We propose a method where the rank adjustment is integrated into the training process, allowing for continuous optimization of model complexity. Our experiments across diverse datasets and network architectures, including CNNs and ANNs, demonstrate ALRA's remarkable potential, consistently outperforming traditional fixed-rank approximations. ALRA's dynamic rank adaptation strategy not only reduces model size but also enhances convergence speed and robustness, making it particularly suited for real-time applications. By introducing ALRA, we offer a novel approach that combines adaptability and low-rank factorization to reshape the landscape of deep learning. We tested ALRA on various datasets like CIFAR-10, MNIST, SVHM, MNIST-Fashion, Iris, Wine and Diabetes. Results obtained after the integration of ALRA in CNNs and ANNs are promising and it outperforms the baseline CNNs and ANNs model.