Efficient Classification of Long Documents via State-Space Models

Peng Lu, Suyuchen Wang, Mehdi Rezagholizadeh, Bang Liu, Ivan Kobyzev

Submission Type: Regular Short Paper

Submission Track: Efficient Methods for NLP

Submission Track 2: NLP Applications

Keywords: Long Document Classification, State Space Models, Efficient NLP

Abstract:

Transformer-based models have achieved state-of-the-art performance on numerous NLP applications. However, long documents which are prevalent in real-world scenarios cannot be efficiently processed by transformers with the vanilla self-attention module due to their quadratic computation complexity and limited length extrapolation ability. Instead of tackling the computation difficulty for self-attention with sparse or hierarchical structures, in this paper, we investigate the use of State-Space Models (SSMs) for long document classification tasks.
We conducted extensive experiments on six long document classification datasets, including binary, multi-class, and multi-label classification, comparing SSMs (with and without pre-training) to self-attention-based models. We also introduce the SSM-pooler model and demonstrate that it achieves comparable performance while being on average 36\% more efficient. Additionally our method exhibits higher robustness to the input noise even in the extreme scenario of 40\%.

Submission Number: 4225