Accelerating the Backpropagation Algorithm by Using NMF-Based Method on Deep Neural Networks

Suhyeon Baek, Akira Imakura, Tetsuya Sakurai, Ichiro Kataoka

Abstract: Backpropagation (BP) is the most widely used algorithm for the training of deep neural networks (DNN) and is also considered a de facto standard algorithm. However, the BP algorithm often requires a lot of computation time, which remains a major challenge. Thus, to reduce the time complexity of the BP algorithm, several methods have been proposed so far, but few do not apply to the BP algorithm. In the meantime, a new DNN algorithm based on nonnegative matrix factorization (NMF) has been proposed, and the algorithm has different convergence characteristics from the BP algorithm. We found that the NMF-based method could lead to rapid performance improvement in DNNs training, and we developed a technique to accelerate the training time of the BP algorithm. In this paper, we propose a novel training method for accelerating the BP algorithm by using an NMF-based algorithm. Furthermore, we present a technique to boost the efficiency of our proposed method by concurrently training DNNs with the BP and NMF-based algorithms. The experimental results indicate that our method significantly improves the training time of the BP algorithm.