Exploring the Hidden Dimension in Accelerating Convolutional Neural Networks

Zhihao Jia, Sina Lin, Charles R. Qi, Alex Aiken

Abstract: DeePa is a deep learning framework that explores parallelism in all parallelizable dimensions to accelerate the training process of convolutional neural networks. DeePa optimizes parallelism at the granularity of each individual layer in the network. We present an elimination-based algorithm that finds an optimal parallelism configuration for every layer. Our evaluation shows that DeePa achieves up to 6.5× speedup compared to state-of-the-art deep learning frameworks and reduces data transfers by up to 23×.

TL;DR: To the best of our knowledge, DeePa is the first deep learning framework that controls and optimizes the parallelism of CNNs in all parallelizable dimensions at the granularity of each layer.

Keywords: Parallelism of Convolutional Neural Networks, Accelerating Convolutional Neural Networks