Long text analysis using sliced recurrent neural networks with breaking point information enrichment

Bo Li, Zehua Cheng, Zhenghua Xu, Wei Ye, Thomas Lukasiewicz, Shikun Zhang

Abstract:

Sliced recurrent neural networks (SRNNs) are the state-ofthe-art efficient solution for long text analysis tasks; however,
their slicing operations inevitably result in long-term dependency loss in lower-level networks and thus limit their accuracy. Therefore, we propose a breaking point information
enrichment mechanism to strengthen dependencies between
sliced subsequences without hindering parallelization. Then,
the resulting BPIE-SRNN model is further extended to a bidirectional model, BPIE-BiSRNN, to utilize the dependency information in not only the previous but also the following contexts. Experiments on four large public real-world datasets
demonstrate that the BPIE-SRNN and BPIE-BiSRNN models always achieve a much better accuracy than SRNNs and
BiSRNNs, while maintaining a superior training efficiency