Why Are Kronecker Products So Effective?

Anonymous

Keywords: kronecker, quaternion, parameter efficient, tensor decomposition, SVD

Abstract: In this blog post we will review the layer scheme proposed by the authors of the paper "Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with $$1/n$$ Parameters", which links quaternion-based Neural Networks with the Kronecker Product, and later explain how the Kronecker Product provides a connection between the paper and a parallel line of research into parameter efficient Neural Networks for Computer Vision based on Singular Value Decomposition.

ICLR Paper: https://openreview.net/forum?id=rcQdycl0zyk