Random Features for Normalization Layers

Rebekka Burkholz

Keywords: deep learning, normalization, random features, finetuning

TL;DR: We derive maximally sparse, random weight distributions which induce perfect conditioning for learning normalization layers only.

Abstract: Can we can reduce the number of trainable parameters of neural networks by freezing a large portion of the initial weights? Training only BatchNorm parameters has shown great experimental promise, yet, the ability to express any potential target network would require a high amount of degrees of freedom. Even with sufficiently many parameters, contemporary optimization algorithms achieve only suboptimal performance. We systematically investigate both issues, expressiveness and trainability, and derive sparse random features which enjoy advantages in both aspects. In contrast to standard initialization approaches, they provably induce a well conditioned learning task and learning dynamics that are equivalent to the standard setting. They are also well aligned with target networks that can be approximated by random lottery tickets, which translates into a reduced bound on the number of required features. We obtain this bound by exploiting the layer-wise permutational invariance of target neurons, which applies to general feature distributions with good target alignment, and thus outline a path towards parameter efficient random features.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 4827