Addressing Token Uniformity in Transformers via Singular Value Transformation

Hanqi Yan, Lin Gui, Wenjie Li, Yulan He

Keywords: token uniformity, pre-trained language model, language processing

TL;DR: We characterise the token uniformity problem commonly observed in the output of transformer-based architectures by the degree of skewness in singular value distributions and propose a singular value transformation function to address the problem.

Abstract: Token uniformity is commonly observed in transformer-based models, in which different tokens share a large proportion of similar information after going through stacked multiple self-attention layers in a transformer. In this paper, we propose to use the distribution of singular values of outputs of each transformer layer to characterise the phenomenon of token uniformity and empirically illustrate that a less skewed singular value distribution can alleviate the token uniformity problem. Base on our observations, we define several desirable properties of singular value distributions and propose a novel transformation function for updating the singular values. We show that apart from alleviating token uniformity, the transformation function should preserve the local neighbourhood structure in the original embedding space. Our proposed singular value transformation function is applied to a range of transformer-based language models such as BERT, ALBERT, RoBERTa and DistilBERT, and improved performance is observed in semantic textual similarity evaluation and a range of GLUE tasks. Our source code is available at https://github.com/hanqi-qi/tokenUni.git.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2208.11790/code)