Classical and Quantum Algorithms for Orthogonal Neural Networks

Jonas Landman, Natansh Mathur, Iordanis Kerenidis

Keywords: orthogonal neural networks, orthogonality, quantum computing, quantum machine learning, quantum deep learning, complexity, quantum computer

Abstract: Orthogonal neural networks have recently been introduced as a new type of neural network imposing orthogonality on the weight matrices. They could achieve higher accuracy and avoid evanescent or explosive gradients for deep architectures. Several classical gradient descent methods have been proposed to preserve orthogonality while updating the weight matrices, but these techniques suffer from long running times and/or provide only approximate orthogonality. In this paper, we introduce a new type of neural network layer called Pyramidal Circuit, which implements an orthogonal matrix multiplication. It allows for gradient descent with perfect orthogonality with the same asymptotic running time as a standard fully connected layer. This algorithm is inspired by quantum computing and can therefore be applied on a classical computer as well as on a near term quantum computer. It could become the building block for quantum neural networks and faster orthogonal neural networks.

One-sentence Summary: A new and fast way of implementing orthogonal neural networks, both for classical and quantum computers, with numerical simulations.