Genetic-based Joint Dynamic Pruning and Learning Algorithm to Boost DNN Performance

Azadeh Famili, Yingjie Lao

Abstract: The learning process of a biological system is a continuous phenomenon with limited external interventions. As learning progress, the numbers of neurons and synapses are modified based on the circumstances, which will impact the learning rate (i.e., learning faster as learning progresses). However, different from the characteristics of biological systems, the current research on deep learning is focused on a fixed training process with a predefined architecture to obtain optimal accuracy. On the other hand, while model pruning techniques have been studied to eliminate redundant neurons or synapses, most of them are applied after training but before deployment to accelerate the inference. In this paper, we integrate pruning into training and propose a genetic-based joint pruning and learning algorithm that monitors the training process and prunes the redundant parameters while training. As a result, our method can accelerate both training and inference. The proposed genetic-based method is well-suited for both training from scratch and online learning tasks by considering both the importance and stability of the parameters in the pruning process. The effectiveness of the proposed algorithm is evaluated on different neural network architectures and datasets, which demonstrates significant improvements for the training under both batch learning and incremental learning scenarios.