One Reflection Suffice

Alexander Mathiasen, Frederik Hvilshøj

Keywords: Orthogonal Weights Householder Reflections Normalizing Flows

Abstract: Orthogonal weight matrices are used in many areas of deep learning. Much previous work attempt to alleviate the additional computational resources it requires to constrain weight matrices to be orthogonal. One popular approach utilizes *many* Householder reflections. The only practical drawback is that many reflections cause low GPU utilization. We mitigate this final drawback by proving that *one* reflection is sufficient, if the reflection is computed by an auxiliary neural network.

One-sentence Summary: Instead of using many Householder reflections you can just use one "auxillary" reflection.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=oowmG3kJXQ