$\mathcal{D}^2$-Sparse: Navigating the low data learning regime with coupled sparse networks
Keywords: sparsity, model pruning, low data learning, model merging, dynamic sparsity
TL;DR: We propose a novel method $\mathcal{D}^2$-Sparse that takes advantage of dynamic pruning and model merging in a coupled network framework that improves performance substantially over baselines in extreme small data fraction learning.
Abstract: Research within the realm of deep learning has extensively delved into learning under diverse constraints, with the incorporation of sparsity as a pragmatic constraint playing a pivotal role in enhancing the efficiency of deep learning. This paper introduces a novel approach, termed $\mathcal{D}^2$-Sparse, presenting a dual dynamic sparse learning system tailored for scenarios involving limited data. In contrast to conventional studies that independently investigate sparsity and low-data learning, our research amalgamates these constraints, paving the way for new avenues in sparsity-related investigations. $\mathcal{D}^2$-Sparse outperforms typical iterative pruning methods when applied to standard deep networks, particularly excelling in tasks like image classification within the domain of computer vision. In particular, it achieves a notable 5\% improvement in top-1 accuracy for ResNet-34 in the CIFAR-10 classification task, with only 5000 samples compared to iterative pruning methods.
Submission Number: 1
Loading