Frontal low-rank random tensors for high-order feature representation

Yan Zhang, Krikamol Muandet, Qianli Ma, Heiko Neumann, Siyu Tang

Abstract: Representing high-order (second-order or higher) information in deep neural nets is essential in many tasks such as fine-grained visual understanding and multi-modal information fusion. Bilinear models are often used to extract second-order information. As a basis, extracting higher-order information requires extra computation. In this paper, we propose an approach to representing high-order information via a simple yet effective bilinear form. Specifically, our contribution is two-fold: (1) From the multilinear perspective, we derive a bilinear form of low complexity, assuming that the three-way tensor has low-rank frontal slices. (2) Rather than learning the tensor entries from data, we sample the entries from different underlying distributions, and prove that the underlying distribution influences the information order. We perform temporal action segmentation experiments to evaluate our method. The results demonstrate that our bilinear form, employed as intermediate layers in deep neural nets, is computationally efficient; meanwhile it is effective as it achieves new state-of-the-art results on public benchmarks.

Code: https://github.com/anonymous-rubick/ms-tcn-bilinear