Intelligent Matrix Exponentiation

Thomas Fischbacher, Iulia Maria Comsa, Krzysztof Potempa, Moritz Firsching, Luca Versari, Jyrki Alakuijala

Keywords: matrix exponential, tensor methods, supervised learning, domain extrapolation, certified robustness

Abstract: We present a novel machine learning architecture that uses a single high-dimensional nonlinearity consisting of the exponential of a single input-dependent matrix. The mathematical simplicity of this architecture allows a detailed analysis of its behaviour, providing robustness guarantees via Lipschitz bounds. Despite its simplicity, a single matrix exponential layer already provides universal approximation properties and can learn and extrapolate fundamental functions of the input, such as periodic structure or geometric invariants. This architecture outperforms other general-purpose architectures on benchmark problems, including CIFAR-10, using fewer parameters.

One-sentence Summary: The matrix exponential as a high-dimensional nonlinearity in machine learning of geometric, periodic and general structures.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2008.03936/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=QpAqiF8TTx