Performance Analysis for Tensor-Train Decomposition to Deep Neural Network Based Vector-to-Vector Regression

Jun Qi, Xiaoli Ma, Chin-Hui Lee, Jun Du, Sabato Marco Siniscalchi

Abstract: This work focuses on a performance analysis of tensor-train decomposition applied to the deep neural network (DNN) based vector-to-vector regression. Tensor-train Network (TTN), obtained through tensor-train decomposition, converts a DNN based vector-to-vector regression into a tensor-to-vector mapping with fewer parameters. We can therefore build an over-parametrized DNN with the tensor-train representation such that the optimization error can be significantly reduced, while the upper bounds on the approximation and estimation errors can be maintained. We compare TTN-based neural architecture against an over-parametrized DNN on the MNIST dataset, and the experimental evidence demonstrates the validity of our conjectures on our proposed performance bounds.