Training Structured Efficient Convolutional Layers

Oct 20, 2018 NIPS 2018 Workshop CDNNRIA Blind Submission readers: everyone
  • Abstract: Typical recent neural network designs are primarily convolutional layers, but the tricks enabling structured efficient linear layers (SELLs) have not yet been adapted to the convolutional setting. We present a method to express the weight tensor in a convolutional layer using diagonal matrices, discrete cosine transforms (DCTs) and permutations that can be optimised using standard stochastic gradient methods. A network composed of such structured efficient convolutional layers (SECL) outperforms existing low-rank networks and demonstrates competitive computational efficiency.
  • TL;DR: It's possible to substitute the weight matrix in a convolutional layer to train it as a structured efficient layer; performing as well as low-rank decomposition.
  • Keywords: convolutional, low-rank, cifar-10, acdc
0 Replies