Fractional Tensor Recurrent Unit (fTRU): A Stable Forecasting Model With Long Memory

Hejia Qiu, Chao Li, Ying Weng, Zhun Sun, Qibin Zhao

Abstract:

The tensor recurrent model is a family of nonlinear
dynamical systems, of which the recurrence relation consists of
a p-fold (called degree- p) tensor product. Despite such models
frequently appearing in advanced recurrent neural networks
(RNNs), to this date, there are limited studies on their long
memory properties and stability in sequence tasks. In this article,
we propose a fractional tensor recurrent model, where the tensor
degree p is extended from the discrete domain to the continuous
domain, so it is effectively learnable from various datasets.
Theoretically, we prove that a large degree p is essential to
achieve the long memory effect in a tensor recurrent model, yet
it could lead to unstable dynamical behaviors. Hence, our new
model, named fractional tensor recurrent unit (fTRU), is expected
to seek the saddle point between long memory property and
model stability during the training. We experimentally show that
the proposed model achieves competitive performance with a
long memory and stable manners in several forecasting tasks
compared to various advanced RNNs.