Residual-time gated recurrent unit

Yirong Wu, Feng Hu, Chonghao Yue, Shuifa Sun

Abstract: Recurrent neural networks (RNNs) are well-suited for sequential data processing, which have been widely used in natural language processing, speech recognition, and other sequential signal recognition tasks. However, for long sequential data processing, as the number of network layers increases, conventional RNNs are prone to problems such as vanishing gradients and weak capacity of memorizing long-term information, leading to reduced accuracy. In this article, we propose a residual-time gated recurrent unit (RT-GRU) based deep learning model to address these problems. The RT-GRU model introduces residual information into the candidate hidden state representation of the GRU in the backpropagation direction, making the network more sensitive to gradient changes. Additionally, it establishes a residual connection in the backpropagation through time direction, allowing the network to capture long-term dependencies. Moreover, the gating mechanism of the highway networks is utilized to control the residual information flow. The results of the adding problem, language modeling, text classification, and sentiment analysis experiments demonstrate that RT-GRU achieves a superior performance over several state-of-the-art baseline methods on some well-known datasets.