Orthogonalising gradients to speedup neural network optimisation

Mark Tuddenham, Adam Prugel-Bennett, Jonathon Hare

Keywords: machine learning, deep learning, orthogonalisation, optmisation, optimization

Abstract:

The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimisation step, ensuring the diversification of the learned representations. We hypothesize that components in the same layer learn the same representations at the beginning of learning. To prevent this we orthogonalise the gradients of the components with respect to each other.
Our method of orthogonalisation allows the weights to be used more flexibly, in contrast to restricting the weights to an orthogonalised sub-space. We tested this method on ImageNet and CIFAR-10 resulting in a large decrease in learning time, and also obtain a speed-up on the semi-supervised learning BarlowTwins. We obtain similar accuracy to SGD without fine-tuning and better accuracy for naïvely chosen hyper-parameters.

One-sentence Summary: The optimisation of neural networks can be sped up by orthogonalising the gradients before the optimiser step.