Go to ICOMP 2024 Conference homepageExpressive Power of Tensor-Train Networks With Equal TT-cores
Deep neural networks show their extreme efficiency in solving a wide range of practical problems. Despite this, a theoretical explanation of this phenomenon is only beginning to emerge in scientific research. For some special kinds of deep neural networks, it has been shown that depth is the key to efficiency. In particular, it has recently been shown that Tensor-Train networks, i.e. recurrent neural networks, each layer of which implements a bilinear function, are exponentially more expressive than shallow networks. However, in practice, recurrent neural networks with identical layers are used, the analogue of which are Tensor-Train networks with equal TT-cores. For this class of networks, the analogous result was not proved, but formulated as a Hypothesis. We prove this Hypothesis and thus close the question of exponential expressivity of traditional recurrent neural networks. We also conduct a series of numerical experiments to confirm the theoretical result.