ZerO Initialization: Initializing Residual Networks with only Zeros and Ones

Jiawei Zhao, Florian Tobias Schaefer, Anima Anandkumar

Keywords: weight initialization, deep residual network, deterministic initialization, optimization

Abstract: Deep neural networks are usually initialized with random weights, with adequately selected initial variance to ensure stable signal propagation during training. However, there is no consensus on how to select the variance, and this becomes challenging especially as the number of layers grows. In this work, we replace the widely used random weight initialization with a fully deterministic initialization scheme ZerO, which initializes residual networks with only zeros and ones. By augmenting the standard ResNet architectures with a few extra skip connections and Hadamard transforms, ZerO allows us to start the training from zeros and ones entirely. This has many benefits such as improving reproducibility (by reducing the variance over different experimental runs) and allowing network training without batch normalization. Surprisingly, we find that ZerO achieves state-of-the-art performance over various image classification datasets, including ImageNet, which suggests random weights may be unnecessary for modern network initialization.

One-sentence Summary: We propose a fully deterministic initialization for training residual networks by employing skip connections and Hadamard transforms, resulting in state-of-art performance.