The Singular Values of Convolutional Layers

Hanie Sedghi, Vineet Gupta, Philip M. Long

Keywords: singular values, operator norm, convolutional layers, regularization

TL;DR: We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation.

Abstract: We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2% to 5.3%.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1805.10408/code)