Exploiting Randomness in Deep Learning Algorithms

Hamed Fatemi Langroudi, Cory E. Merkel, Humza Syed, Dhireesha Kudithipudi

Abstract: The recent surge of interest in using deep neural networks for real-world tasks has led to training complex networks with billions of parameters that use enormous amounts of training data. Performing backpropagation in these deep networks is time consuming and requires large amount of resources that is usually limited by the underlying hardware. In order to move towards agile deep learning, we are motivated to exploit randomness in the networks. In this work, we explore the effects of utilizing random weights in convolutional neural networks. This is achieved through random initialization of weights and by freezing them. The training occurs only in the output layer. We also propose a novel weight distribution method based on the sum of sinusoids for random convolutional neural networks. Our experiments show that by leaving the weights random in convolutional neural networks relatively high performance can be achieved for MSTAR and CIFAR-10 datasets.